Add automated backup and restore tests #3

pcahyna · 2023-09-01T10:27:31Z

This PR adds two Beakerlib ( https://github.com/beakerlib/beakerlib/wiki/man ) tests for the complete unattended backup and restore process:

make-backup-and-restore-bios, by @lzaoral. This tests backup and restore using the USB method, backup is stored to a second disk drive, which is also used to boot the rescue system from. This test can not be executed automatically in CI until the test infrastructure we are using (Testing Farm) supports adding a second disk to the VMs where the tests are executing. It serves as a basis for the second test:
make-backup-and-restore-iso, by @antonvoznia. This uses the ISO output. A ramdisk is used to emulate the CD-ROM drive. The ISO image with backup gets written to the ramdisk after ReaR recovery starts and before it starts reformatting the disk (where the ISO image is stored) to ensure that the backup persists until we need to restore it. The changes relative to make-backup-and-restore-bios are described in "A. Voznia: Automation of testing of operating system backup and recovery (2023)" https://dspace.cvut.cz/handle/10467/107238.

Solves one part of rear/rear#3014 (tests will need to enabled in the rear repository in order to get executed in CI).

Will serve as a basis for a new test using a virtual CD drive instead of a second disk.

Can be changed if ReaR is being run from a non-default location (git checkout for example).

gdha · 2023-09-01T12:48:56Z

@pcahyna you should now have the required permissions to merge the PR

lzaoral

I've tried the make-backup-and-restore-iso test locally and it works!

Nitpicks:

Outdated years in copyright.
You probably want to update authors in the remaining files in the make-backup-and-restore-iso directory.
I'd squash the make-backup-and-restore-iso related commits before merging.

I'd add more comments what is the make-backup-and-restore-iso test actually trying to achieve.
While it is great that it works, I have no idea how and why are the memdisk shenanigans required.

lzaoral · 2023-09-04T11:04:21Z

tests/make-backup-and-restore-iso/runtest.sh

+ISO_DEFAULT=automatic
+ISO_RECOVER_MODE=unattended' | tee $REAR_CONFIG" \


These changes are included in 4cafe30 but do not seem to be related to it at all.

lzaoral · 2023-09-04T11:05:03Z

tests/make-backup-and-restore-iso/runtest.sh

@@ -73,13 +73,15 @@ rlJournalStart
        rlPhaseEnd

        rlPhaseStartSetup "Create /etc/rear/local.conf"


You probably want to use $REAR_CONFIG here as well?

lzaoral · 2023-09-04T11:07:21Z

tests/make-backup-and-restore-iso/runtest.sh

-            rlRun "rm -f drive_layout.{old,new}" 0 "Remove lsblk outputs"
-            rlRun "rm -rf /root/rear*.log /var/log/rear/*" 0 "Remove ReaR logs"
+            rlRun "rm -f $REAR_HOME_DIRECTORY/drive_layout.{old,new}" 0 "Remove lsblk outputs"
+            rlRun "rm -rf rear*.log /var/log/rear/*" 0 "Remove ReaR recovery log"


Will the plain rear*.log glob remove correct files? Few lines above, the same file as referenced with the $REAR_HOME_DIRECTORY prefix. (Also, the message change seems unnecessary.)

I believe now that rear*.log should be always referenced as /root/rear*.log and not as $REAR_HOME_DIRECTORY/rear*.log. So, few lines above it is wrong. This is because this log is created by ReaR recovery and it creates it under /root unconditionally (more precisely, under $ROOT_HOME_DIR, which is set to ~root by default). So it is not affected by our $REAR_HOME_DIRECTORY setting.

@lzaoral the last commit should resolve all the $REAR_HOME_DIRECTORY confusion ...

lzaoral · 2023-09-04T11:10:15Z

tests/make-backup-and-restore-iso/runtest.sh

+            rlLog "Setup GRUB"
+            rlRun "echo 'menuentry \"ReaR-recover\" {
+  loopback loop (hd0,msdos2)/root/rear/var/lib/rear/output/rear-fedora.iso
+  linux (loop)/isolinux/kernel rw selinux=0 console=ttyS0,9600 console=tty0 auto_recover unattended


Are you sure that we will always use these hard-coded console devices? What was wrong with the detection implemented in therlPhaseStartSetup "Redirect ReaR output to $CONSOLE_DEVICE" phase which was removed?

It looks like next commits do not even need these parameters, right?

lzaoral · 2023-09-04T11:13:17Z

tests/make-backup-and-restore-iso/runtest.sh

+            rlLog "Setup GRUB"
+            rlRun "echo 'menuentry \"ReaR-recover\" {
+  loopback loop (hd0,msdos2)/root/rear/var/lib/rear/output/rear-fedora.iso
+  linux (loop)/isolinux/kernel rw selinux=0 console=ttyS0,9600 console=tty0 auto_recover unattended


Also, is it intended that we boot the recovery directly from GRUB? I know that it has some issues with chainloading but it would be nice to also test the bootloader (ISOLINUX AFAIK) stored on the ISO image.

Ok, it looks like next commits have done exactly that...

lzaoral · 2023-09-04T11:15:50Z

tests/make-backup-and-restore-iso/runtest.sh

@@ -60,9 +60,12 @@ check_and_submit_rear_log() {
    rlFileSubmit $path "rear-$1.log"
 }

+ROOT_DISK=$(df -hT | grep /$ | awk '{print $1}')


If this works reliably enough then fine, but device naming in linux is not stable.

edit: typo

lzaoral · 2023-09-04T11:16:05Z

tests/make-backup-and-restore-iso/runtest.sh

+        rlPhaseStartSetup "Make small iso file that is bootable by memdisk"
+            rlRun "xorriso -as mkisofs -r -V 'REAR-ISO' -J -J -joliet-long -cache-inodes -b isolinux/isolinux.bin -c isolinux/boot.cat -boot-load-size 4 -boot-info-table -no-emul-boot -eltorito-alt-boot -dev $REAR_ISO_OUTPUT/rear-$HOSTNAME_SHORT.iso -o $REAR_ISO_OUTPUT/small-rear.iso -- -rm_r backup"
+        rlPhaseEnd
+


Nitpick: two empty lines.

lzaoral · 2023-09-04T11:16:40Z

tests/make-backup-and-restore-iso/runtest.sh

+            rlLog "Copying memdisk"
+            rlRun "cp /usr/share/syslinux/memdisk /boot/"


These commands can be combined because rlRun can specify comments.

lzaoral · 2023-09-04T11:16:58Z

tests/make-backup-and-restore-iso/runtest.sh

+            rlLog "Setup GRUB"
+            rlRun "echo 'menuentry \"ReaR-recover\" {


These commands can be combined because rlRun can specify comments.

pcahyna · 2023-09-10T16:34:59Z

@lzaoral I believe I addressed most of your comments. Please check whether the comments added to the script are explanatory enough.

lzaoral

Thanks for the update, @pcahyna! I was not able to successfully execute the test using tmt on Fedora Rawhide (it might be useful to also add a default test plan in the future) even after applying the following patch:

diff --git a/tests/make-backup-and-restore-iso/main.fmf b/tests/make-backup-and-restore-iso/main.fmf
index 286bb23..4db7762 100644
--- a/tests/make-backup-and-restore-iso/main.fmf
+++ b/tests/make-backup-and-restore-iso/main.fmf
@@ -1,6 +1,7 @@
 summary: 'Test basic functionality of ReaR on systems with BIOS using bootable ISO image'

 require:
+    - rear
     - syslinux-extlinux
     - syslinux-nonlinux
     - xorriso

The test timeouted after a while. Direct execution though 1mt also ended with a failure during boot of the rescue environment. Very small nitpick below. Also, I'd still squash the commits that lead to the final version of make-backup-and-restire-iso before merging.

tests/make-backup-and-restore-iso/runtest.sh

lzaoral · 2023-09-26T11:14:55Z

As noted by @pcahyna in a private conversation, the test (or more probably ReaR) is not yet compatible with Fedora Rawhide. I was able to successfully execute the test both using tmt and 1mt on a RHEL 9 machine with a sufficiently large amount of RAM.

pcahyna · 2023-09-26T12:14:52Z

@lzaoral the test plan will be in the ReaR repository: pcahyna/rear#13 . It also includes information about the excluded Fedora versions and the reason behind.

REAR_BIN = path to the ReaR executable REAR_CONFIG = the configuration file (local.conf) REAR_HOME_DIRECTORY = directory with various test outputs Co-authored-by: Pavel Cahyna <pcahyna@redhat.com> Co-authored-by: Lukáš Zaoral <lukaszaoral@outlook.com>

Co-authored-by: antonvoznia <antoncty@gmail.com>

This uses the ISO output. A ramdisk is used to emulate the CD-ROM drive. The ISO image with backup gets written to the ramdisk after ReaR recovery starts and before it starts reformatting the disk (where the ISO image is stored) to ensure that the backup persists until we need to restore it. The changes relative to make-backup-and-restore-bios are described in "A. Voznia: Automation of testing of operating system backup and recovery (2023)" https://dspace.cvut.cz/handle/10467/107238. Co-authored-by: Pavel Cahyna <pcahyna@redhat.com>

Delete hardcoded hostname, get it from hostname -s Get disk's name of root partition Add rear to tmt metadata requirements Co-authored-by: Pavel Cahyna <pcahyna@redhat.com> Co-authored-by: Lukáš Zaoral <lukaszaoral@outlook.com>

Use relative path to boot partition Find boot partition via GRUB search Rename iso without backup to be more clear

pcahyna · 2023-10-03T15:50:44Z

I squashed and reworded commits without any content changes. Merging.

pcahyna · 2023-11-18T18:39:32Z

As noted by @pcahyna in a private conversation, the test (or more probably ReaR) is not yet compatible with Fedora Rawhide.

Solved in PR rear/rear#3079

lzaoral and others added 4 commits August 31, 2023 17:10

BIOS: Add make-backup-and-restore-bios test

f9dad8b

copy make-backup-and-restore-bios

4ba61a4

Will serve as a basis for a new test using a virtual CD drive instead of a second disk.

fmf init

65e0abe

Set parameters for ReaR location

980d662

Can be changed if ReaR is being run from a non-default location (git checkout for example).

pcahyna mentioned this pull request Sep 1, 2023

RFC: enabling automatic backup and restore test rear/rear#3014

Closed

gdha self-requested a review September 1, 2023 12:45

gdha assigned gdha and pcahyna Sep 1, 2023

gdha added the enhancement label Sep 1, 2023

gdha approved these changes Sep 1, 2023

View reviewed changes

lzaoral suggested changes Sep 4, 2023

View reviewed changes

pcahyna force-pushed the test-backup-recover branch from 6c483ca to 0e7411a Compare September 8, 2023 15:36

lzaoral suggested changes Sep 25, 2023

View reviewed changes

tests/make-backup-and-restore-iso/runtest.sh Outdated Show resolved Hide resolved

antonvoznia and others added 13 commits October 3, 2023 17:27

Add parameters for unattended ISO recovery

e3b07e7

Co-authored-by: antonvoznia <antoncty@gmail.com>

Do not use an unavailable and unneeded script

9a06f4c

Create a smaller iso to boot via memdisk

28e67cd

Delete hardcoded hostname, get it from hostname -s Get disk's name of root partition Add rear to tmt metadata requirements Co-authored-by: Pavel Cahyna <pcahyna@redhat.com> Co-authored-by: Lukáš Zaoral <lukaszaoral@outlook.com>

Delete hardcoded boot and root relative pathes

e8ae929

Use relative path to boot partition Find boot partition via GRUB search Rename iso without backup to be more clear

Disable ssh connection in rear rescue recover mode

c1971a0

Use /var/tmp for backup storage

51ecd2e

Increase timeout for reboot

bfca981

Add author in copyright

013d948

Sync Requires in iso test Makefile

7a6dba1

Update authors, copyright, test name, etc.

619aea9

Add explanation of ISO handling and bootloader use

3d3d3a8

pcahyna force-pushed the test-backup-recover branch from e4e6ff2 to 3d3d3a8 Compare October 3, 2023 15:47

pcahyna merged commit d00d35c into rear:master Oct 3, 2023

pcahyna mentioned this pull request Oct 4, 2023

Enable automated backup & recovery test in CI on PRs rear/rear#3052

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add automated backup and restore tests #3

Add automated backup and restore tests #3

pcahyna commented Sep 1, 2023

gdha commented Sep 1, 2023

lzaoral left a comment

lzaoral Sep 4, 2023

lzaoral Sep 4, 2023

lzaoral Sep 4, 2023

pcahyna Sep 12, 2023

pcahyna Sep 13, 2023

lzaoral Sep 4, 2023

lzaoral Sep 4, 2023

lzaoral Sep 4, 2023

lzaoral Sep 4, 2023

lzaoral Sep 4, 2023 •

edited

lzaoral Sep 4, 2023

lzaoral Sep 4, 2023

lzaoral Sep 4, 2023

pcahyna commented Sep 10, 2023

lzaoral left a comment

lzaoral commented Sep 26, 2023

pcahyna commented Sep 26, 2023

pcahyna commented Oct 3, 2023

pcahyna commented Nov 18, 2023 •

edited

		ISO_DEFAULT=automatic
		ISO_RECOVER_MODE=unattended' \| tee $REAR_CONFIG" \

		@@ -73,13 +73,15 @@ rlJournalStart
		rlPhaseEnd

		rlPhaseStartSetup "Create /etc/rear/local.conf"

		rlLog "Copying memdisk"
		rlRun "cp /usr/share/syslinux/memdisk /boot/"

Add automated backup and restore tests #3

Add automated backup and restore tests #3

Conversation

pcahyna commented Sep 1, 2023

gdha commented Sep 1, 2023

lzaoral left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lzaoral Sep 4, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pcahyna commented Sep 10, 2023

lzaoral left a comment

Choose a reason for hiding this comment

lzaoral commented Sep 26, 2023

pcahyna commented Sep 26, 2023

pcahyna commented Oct 3, 2023

pcahyna commented Nov 18, 2023 • edited

lzaoral Sep 4, 2023 •

edited

pcahyna commented Nov 18, 2023 •

edited