SUSE 12.2 ReaR recovery stuck at system layout restoration #1786
Comments
|
@manums1983 I'm afraid that from such vague problem description, none will be actually able to help you. Thanks V. |
|
Hi Vladimir,
Thank you for your response, ok I will do the recovery using the command “rear -d -D recover” . Now I am able to login to the server. Please find the details requested below. I am using a file based backup to local mount point “/mnt/backup”. I noticed this /mnt/backup is mounted in a btrfs file system. Now I deleted the volume and recreated with ext4 file system. Running the backup again now after this I will do a restore again.
################local.conf #################################################
# Begin example setup for SLE12-SP2 with default btrfs subvolumes.
# Since SLE12-SP1 what is mounted at '/' is a btrfs snapshot subvolume
# see #556
# and since SLE12-SP2 btrfs quota via "snapper setup-quota" is needed
# see #999
# You must adapt "your.NFS.server.IP/path/to/your/rear/backup" at BACKUP_URL.
# You must decide whether or not you want to have /home/* in the backup.
# It depends on the size of your harddisk whether or not /home is by default
# a btrfs subvolume or a separated xfs filesystem on a separated partition.
# You may activate SSH_ROOT_PASSWORD and adapt the "password_on_the_rear_recovery_system".
# For basic information see the SLE12-SP2 manuals.
# Also see the support database article "SDB:Disaster Recovery"
# at http://en.opensuse.org/SDB:Disaster_Recovery
# In particular note:
# There is no such thing as a disaster recovery solution that "just works".
# Regarding btrfs snapshots:
# Recovery of btrfs snapshot subvolumes is not possible.
# Only recovery of "normal" btrfs subvolumes is possible.
# On SLE12-SP1 and SP2 the only exception is the btrfs snapshot subvolume
# that is mounted at '/' but that one is not recreated but instead
# it is created anew from scratch during the recovery installation with the
# default first btrfs snapper snapshot subvolume path "@/.snapshots/1/snapshot"
# by the SUSE tool "installation-helper --step 1" (cf. below).
# Other snapshots like "@/.snapshots/234/snapshot" are not recreated.
# Create rear recovery system as ISO image:
OUTPUT=ISO
# Store the backup file via NFS on a NFS server:
BACKUP=NETFS
# BACKUP_OPTIONS variable contains the NFS mount options and
# with 'mount -o nolock' no rpc.statd (plus rpcbind) are needed:
BACKUP_OPTIONS="nfsvers=3,nolock"
# If the NFS server is not an IP address but a hostname,
# DNS must work in the rear recovery system when the backup is restored.
BACKUP_URL=file:///mnt/backup
# Keep an older copy of the backup in a HOSTNAME.old directory
# provided there is no '.lockfile' in the HOSTNAME directory:
NETFS_KEEP_OLD_BACKUP_COPY=10
# Have all modules of the original system in the recovery system with the
# same module loading ordering as in the original system by using the output of
# lsmod | tail -n +2 | cut -d ' ' -f 1 | tac | tr -s '[:space:]' ' '
# as value for MODULES_LOAD (cf. #626):
#MODULES_LOAD=( )
# On SLE12-SP1 and SP2 with default btrfs subvolumes what is mounted at '/' is a btrfs snapshot subvolume
# that is controlled by snapper so that snapper is needed in the recovery system.
# In SLE12-SP1 and SP2 some btrfs subvolume directories (/var/lib/pgsql /var/lib/libvirt/images /var/lib/mariadb)
# have the "no copy on write (C)" file attribute set so that chattr is required in the recovery system
# and accordingly also lsattr is useful to have in the recovery system (but not strictly required):
REQUIRED_PROGS=( "${REQUIRED_PROGS[@]}" snapper chattr lsattr )
# Snapper setup by the recovery system uses /usr/lib/snapper/installation-helper
# that is linked to all libraries where snapper is linked to
# (except libdbus that is only needed by snapper).
# "installation-helper --step 1" creates a snapper config based on /etc/snapper/config-templates/default
COPY_AS_IS=( "${COPY_AS_IS[@]}" /usr/lib/snapper/installation-helper /etc/snapper/config-templates/default )
# Files in btrfs subvolumes are excluded by 'tar --one-file-system'
# so that such files must be explicitly included to be in the backup.
# Files in the following SLE12-SP2 default btrfs subvolumes are
# in the below example not included to be in the backup
# /.snapshots /var/crash
BACKUP_PROG_EXCLUDE=( "${BACKUP_PROG_EXCLUDE[@]}" "/media/backup" "/var/crash" "/export/archive" "/home/oracle" "/oracle" "/home/oraprd" "/export/Doc" "/usr/sap/hostctrl" "/sapmnt/PRD" "/usr/sap/SMD" "/usr/sap/tmp" "/usr/sap/PRD" "/sapmnt/PRD/exe" "/oracle/stage" "/oracle/PRD" "/oracle/client" "/oracle/oraprd" "/oracle/PRD/12102" "/oracle/PRD/origlogA" "/oracle/PRD/mirrlogA" "/oracle/PRD/mirrlogB" "/oracle/PRD/origlogB" "/oracle/PRD/sapreorg" "/oracle/PRD/sapdata3" "/oracle/PRD/oraarch" "/oracle/PRD/sapdata4" "/oracle/PRD/sapdata6" "/oracle/PRD/sapdata2" "/oracle/PRD/sapdata1" "/oracle/PRD/sapdata5" "/mnt/backup" )
EXCLUDE_RECREATE=( "${EXCLUDE_RECREATE[@]}" "fs:/media/backup" "fs:/var/crash" "fs:/export/archive" "fs:/home/oracle" "fs:/oracle" "fs:/home/oraprd" "fs:/export/Doc" "fs:/usr/sap/hostctrl" "fs:/sapmnt/PRD" "fs:/usr/sap/SMD" "fs:/usr/sap/tmp" "fs:/usr/sap/PRD" "fs:/sapmnt/PRD/exe" "fs:/oracle/stage" "fs:/oracle/PRD" "fs:/oracle/client" "fs:/oracle/oraprd" "fs:/oracle/PRD/12102" "fs:/oracle/PRD/origlogA" "fs:/oracle/PRD/mirrlogA" "fs:/oracle/PRD/mirrlogB" "fs:/oracle/PRD/origlogB" "fs:/oracle/PRD/sapreorg" "fs:/oracle/PRD/sapdata3" "fs:/oracle/PRD/oraarch" "fs:/oracle/PRD/sapdata4" "fs:/oracle/PRD/sapdata6" "fs:/oracle/PRD/sapdata2" "fs:/oracle/PRD/sapdata1" "fs:/oracle/PRD/sapdata5" "fs:/mnt/backup" )
# but files in /home are included to be in the backup.
# You may use a command like
# findmnt -n -r -o TARGET -t btrfs | grep -v '^/$' | egrep -v 'snapshots|crash'
# to generate the values:
BACKUP_PROG_INCLUDE=( /var/cache /var/lib/mailman /var/tmp /var/lib/pgsql /usr/local /opt /var/lib/libvirt/images /boot/grub2/i386-pc /var/opt /srv /boot/grub2/x86_64-efi /var/lib/mariadb /var/spool /var/lib/mysql /tmp /home /var/log /var/lib/named /var/lib/machines )
# The following POST_RECOVERY_SCRIPT implements during "rear recover"
# btrfs quota setup for snapper if that is used in the original system:
POST_RECOVERY_SCRIPT=( 'if snapper --no-dbus -r $TARGET_FS_ROOT get-config | grep -q "^QGROUP.*[0-9]/[0-9]" ; then snapper --no-dbus -r $TARGET_FS_ROOT set-config QGROUP= ; snapper --no-dbus -r $TARGET_FS_ROOT setup-quota && echo snapper setup-quota done || echo snapper setup-quota failed ; else echo snapper setup-quota not used ; fi' )
# This option defines a root password to allow SSH connection
# whithout a public/private key pair
#SSH_ROOT_PASSWORD="password_on_the_rear_recovery_system"
# Let the rear recovery system run dhclient to get an IP address
# instead of using the same IP address as the original system:
#USE_DHCLIENT="yes"
# End example setup for SLE12-SP2 with default btrfs subvolumes.
#########################################################################
NAME="SLES"
VERSION="12-SP2"
VERSION_ID="12.2"
PRETTY_NAME="SUSE Linux Enterprise Server 12 SP2"
ID="sles"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:suse:sles:12:sp2"
###################################################
Linux GESPRD1 4.4.103-92.56-default #1 SMP Wed Dec 27 16:24:31 UTC 2017 (2fd2155) x86_64 x86_64 x86_64 GNU/Linux
#######################################################
Relax-and-Recover 2.3-git.0.dd982eb.unknown / 2018-04-16
#################################################
Thanks & Regards
Manu MS
Technology Consultant
HPE Pointnext
Mobile: +6591747535
Email : manu.ms@hpe.com<mailto:manu.ms@hpe.com>
From: Vladimir Gozora [mailto:notifications@github.com]
Sent: Friday, April 27, 2018 2:40 PM
To: rear/rear <rear@noreply.github.com>
Cc: MOHANAN SATHYKUMARI, MANU <manu.mohanan-sathykumari@hpe.com>; Mention <mention@noreply.github.com>
Subject: Re: [rear/rear] SUSE 12.2 ReaR recovery stuck at system layout restoration (#1786)
@manums1983<https://github.com/manums1983> I'm afraid that from such vague problem description, none will be actually able to help you.
I'm suspecting that you are doing rear recover.
Can you relaunch with rear -d -D recover and provide us with log files from /var/log/rear ?
Thanks
V.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<#1786 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/Ak-0Cy7gPNKA1CrLSYA4ajGr7wbFQjftks5tsr0tgaJpZM4Tp12D>.
|
|
My humble recommendation would be "Do not use backup to local filesystem (file://) until you are 100% sure what you are doing and what the implications are". Try to go with nfs:// or smb:// for the start. V. |
|
HI Vladimir,
Thank for the information about the file based backup implications. We have tested SUSE 12.2 on another server using file based backup and restore it works fine which is on hardware HPE GEN9 DL380. Since we do not have an NFS server we need to use file based backup and recovery.
Current hardware is HPE GEN9 DL 580, it is a SAP server. Started the restore with command “rear -d -D recover” still it is stuck at “Start System Layout Restoration” I have attached some screenshots and backup logs. Please let know what would be causing this issue.
How to stop the restore. It is in hung state.
Thanks & Regards
Manu MS
From: Vladimir Gozora [mailto:notifications@github.com]
Sent: Friday, April 27, 2018 3:13 PM
To: rear/rear <rear@noreply.github.com>
Cc: MOHANAN SATHYKUMARI, MANU <manu.mohanan-sathykumari@hpe.com>; Mention <mention@noreply.github.com>
Subject: Re: [rear/rear] SUSE 12.2 ReaR recovery stuck at system layout restoration (#1786)
My humble recommendation would be "Do not use backup to local filesystem (file://<file:///\\>) until you are 100% sure what you are doing and what the implications are". Try to go with nfs:// or smb:// for the start.
V.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<#1786 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/Ak-0CwALFVE43y4jkbg49KH5Ye03wpNHks5tssUKgaJpZM4Tp12D>.
|
|
rear case log .docx |
|
I just noticed the /tmp directory got full. /tmp is only 10 GB. Is there a way i can point rear log to a different directory? |
|
@manums1983 I will not even try to read any kind of "proprietary" file format like doc or docx FYI: |
|
FYI: OUTPUT=ISO
BACKUP=NETFS
BACKUP_OPTIONS="nfsvers=3,nolock"
BACKUP_URL=file:///mnt/sdb1
OUTPUT_URL=nfs://10.160.4.244/nfs
REQUIRED_PROGS=( "${REQUIRED_PROGS[@]}" snapper chattr lsattr )
COPY_AS_IS=( "${COPY_AS_IS[@]}" /usr/lib/snapper/installation-helper /etc/snapper/config-templates/default )
BACKUP_PROG_INCLUDE=( /srv /var/lib/mailman /tmp /var/lib/libvirt/images /var/opt /var/log /boot/grub2/i386-pc /var/lib/mysql /var/tmp /opt /boot/grub2/x86_64-efi /var/spool /var/lib/pgsql /var/lib/mariadb /usr/local /home /var/lib/machines /var/lib/named /var/cache )
POST_RECOVERY_SCRIPT=( 'if snapper --no-dbus -r $TARGET_FS_ROOT get-config | grep -q "^QGROUP.*[0-9]/[0-9]" ; then snapper --no-dbus -r $TARGET_FS_ROOT set-config QGROUP= ; snapper --no-dbus -r $TARGET_FS_ROOT setup-quota && echo snapper setup-quota done || echo snapper setup-quota failed ; else echo snapper setup-quota not used ; fi' )
SSH_ROOT_PASSWORD="rear"
USE_DHCLIENT="yes"
but one must know what On the original system I did the preparation steps # parted -s /dev/sdb mklabel msdos # parted -s /dev/sdb unit MiB mkpart primary 1 2047 # mkfs.ext4 /dev/sdb1 # parted -s /dev/sdb unit MiB print Model: ATA QEMU HARDDISK (scsi) Disk /dev/sdb: 2048MiB Sector size (logical/physical): 512B/512B Partition Table: msdos Disk Flags: Number Start End Size Type File system Flags 1 1.00MiB 2047MiB 2046MiB primary ext4 type=83 # mkdir /mnt/sdb1 # mount /dev/sdb1 /mnt/sdb1 and after "rear -D mkbackup" I got # find /mnt/sdb1 /mnt/sdb1 /mnt/sdb1/lost+found /mnt/sdb1/f121 /mnt/sdb1/f121/backup.log /mnt/sdb1/f121/backup.tar.gz and on my NFS server ( nfs-server:/nfs # ls -lhrt /nfs/f121 total 85M -rw------- 1 nobody nogroup 80M Apr 27 10:40 rear-f121.iso -rw------- 1 nobody nogroup 261 Apr 27 10:40 VERSION -rw------- 1 nobody nogroup 202 Apr 27 10:40 README -rw------- 1 nobody nogroup 5.2M Apr 27 10:40 rear-f121.log For the recovery I set up a second virtual machine In the running ReaR recovery system on the second virtual machine # mkdir /mnt/sdb1 # mount /dev/sdb1 /mnt/sdb1 and then I run "rear -D recover" which then "just works" for me. But personally I find using Personally I would perfer OUTPUT=USB to get the backup together I think - with probability one (https://en.wikipedia.org/wiki/Almost_surely) - this issue |
|
Bad typo correction in |
|
Hi Valdimir
I am collecting the restore log file will update you shortly.
Below showing the backup logs
##########################################Backup Successful ###########################
GESPRD1:/mnt/backup # rear -d -D mkbackup
Relax-and-Recover 2.3-git.0.dd982eb.unknown / 2018-04-16
Using log file: /var/log/rear/rear-GESPRD1.log
Using backup archive '/mnt/backup/GESPRD1/backup.tar.gz'
Using UEFI Boot Loader for Linux (USING_UEFI_BOOTLOADER=1)
Creating disk layout
Excluding component fs:/media/backup
Excluding component fs:/var/crash
Excluding component fs:/export/archive
Excluding component fs:/home/oracle
Excluding component fs:/oracle
Excluding component fs:/home/oraprd
Excluding component fs:/export/Doc
Excluding component fs:/usr/sap/hostctrl
Excluding component fs:/sapmnt/PRD
Excluding component fs:/usr/sap/SMD
Excluding component fs:/usr/sap/tmp
Excluding component fs:/usr/sap/PRD
Excluding component fs:/sapmnt/PRD/exe
Excluding component fs:/oracle/stage
Excluding component fs:/oracle/PRD
Excluding component fs:/oracle/client
Excluding component fs:/oracle/oraprd
Excluding component fs:/oracle/PRD/12102
Excluding component fs:/oracle/PRD/origlogA
Excluding component fs:/oracle/PRD/mirrlogA
Excluding component fs:/oracle/PRD/mirrlogB
Excluding component fs:/oracle/PRD/origlogB
Excluding component fs:/oracle/PRD/sapreorg
Excluding component fs:/oracle/PRD/sapdata3
Excluding component fs:/oracle/PRD/oraarch
Excluding component fs:/oracle/PRD/sapdata4
Excluding component fs:/oracle/PRD/sapdata6
Excluding component fs:/oracle/PRD/sapdata2
Excluding component fs:/oracle/PRD/sapdata1
Excluding component fs:/oracle/PRD/sapdata5
Excluding component fs:/mnt/backup
Using sysconfig bootloader 'grub2-efi'
Creating root filesystem layout
Handling network interface 'bond0'
bond0 is a bond
bond0 has lower interface eth0
eth0 is a physical device
bond0 has lower interface eth1
eth1 is a physical device
bond0 has lower interface eth2
eth2 is a physical device
bond0 has lower interface eth3
eth3 is a physical device
Handled network interface 'bond0'
Trying to find what to use as UEFI bootloader...
Trying to find a 'well known file' to be used as UEFI bootloader...
Using '/boot/efi/EFI/sles/grubx64.efi' as UEFI bootloader file
Copying logfile /var/log/rear/rear-GESPRD1.log into initramfs as '/tmp/rear-GESPRD1-partial-2018-04-27T14:40:31+08:00.log'
Copying files and directories
Copying binaries and libraries
Copying kernel modules
Copying all files in /lib*/firmware/
Creating recovery/rescue system initramfs/initrd initrd.cgz with gzip default compression
Created initrd.cgz with gzip default compression (207576391 bytes) in 76 seconds
Making ISO image
Wrote ISO image: /var/lib/rear/output/rear-GESPRD1.iso (242M)
Copying resulting files to file location
Saving /var/log/rear/rear-GESPRD1.log as rear-GESPRD1.log to file location
Creating tar archive '/mnt/backup/GESPRD1/backup.tar.gz'
Archived 12648 MiB [avg 6759 KiB/sec] OK
Archived 12648 MiB in 1917 seconds [avg 6756 KiB/sec]
Exiting rear mkbackup (PID 9207) and its descendant processes
Running exit tasks
You should also rm -Rf /tmp/rear.u7WHb4sgvsPoLPy
Thanks & Regards
Manu MS
|
|
In the /mnt/backup/GESPRD1 will have all gzip file and ISO file. I will copy ISO to my local laptop and attach this ISO to iLO console and boot from rear ISO. GESPRD1:/mnt/backup/GESPRD1 # ls -l |
|
Please find the logs attached. |
|
Hello @manums1983
V. |
|
As I've read your disklayout.conf I've noticed that point 2. of my #1786 (comment) is true. Your /mnt/backup is located on /dev/sdb1 which is SmarArray logical volume spread across single disk. If this is a test server all you have to do before starting it is just a guess, but ReaR might try to re-create your SmartArray configuration with this line active, which is not something you want. If this is however production server, you should reconsider your bare metal disaster recovery strategy, because storing backups locally on SmartArray without any redundancy will really not help you in case of disaster. V. |
jsmeix
added
the
special hardware or VM
|
@gozora |
|
@jsmeix yes, you are right with one tiny correction, it is HPE nowadays :-), this is however not the crux of the problem, but rather (as you already stated)
So I personally would avoid using "special hardware" label ... V. |
jsmeix
removed
the
special hardware or VM
|
Hardware Specs: HPE iLO4 "Smart Array P830i". Hard disk Configuration : RAID1 Array (2x600 GB) (/dev/sda) ---> This is local disk where OS is installed. All other disk are 3PAR multipath disks. All 3PAR disks exclude from the backup and recreation. |
|
I don't know about Smart Array but assume there is real hardware RAID here and @manums1983 |
Correct ;-) |
I'm afraid that there is no replacement server, SmartArray disks are presented to system as any other disk /dev/sda, /dev/sdb ... So I guess that strategy here is just to boot ReaR recovery system and restore content of /dev/sda from /dev/sdb V. |
|
This is purely a hardware RAID array controller. Not software based. This is a production SAP server in DR site. Highly critical to make any kind of changes, Here the difference is DL 580 with OS SUSE 12.2 with lots of 3PAR Luns. Noticed the restore getting stuck at "Start System Layout Restoration" , do we have any issue with multi-path. I wonder how you access it from a replacement server? ---> OS is running on RAID1 (sda) if it corrupted after patching or any other reasons can restore the (sda) from the backup (sdb). Boot from ReaR ISO , in rescue mode mount /dev/sab1 /mnt/backup and start recovery. |
|
correction: |
|
@manums1983 if you think you are heaving problem with multipath and you are doing recovery solely on local disks, just try to unload dm-multipath and friends kernel modules + whatever FC (lpfc, bnx, qlaxxx) driver you are using prior V. |
|
@gozora, |
|
@manums1983 if this is an critical / production server and downtime would mean any kind of SLA breach you should either escalate this task to higher support level or ask some more experienced colleague for help. V. |
|
Hi gozora, |
|
One of the options would be to run recovery with debug options V. |
|
sure i will share you the log on Monday. |
|
Hi Gozora, I like to try backup to USB. Below configuration is ok to boot form USB and do recovery. pls let know. |
|
I try backup to USB. But getting following error in writing logs. Trying to find what to use as UEFI bootloader... |
|
Noticed that the /dev/sdaq1 is getting full and it is failing. |
|
Hi Gozora, ###############Logs################################ ++ StopIfError 'You must specify either BACKUP_URL or BACKUP_MOUNTCMD and BACKUP_UMOUNTCMD !' |
|
Again, if you are heaving trouble during restore phase, please provide log files created by V. |
|
Hi Gozora, |
|
I tried it copy the /var/log/rear/gesprd*** to a local drive. but the server is getting hung. Not responding to any abort commands. |
|
I have attached restore -d -D recover error screenshot. i could not copy any logs to local directory because the whole server is hung. Really appreciate if you can see something form the screenshots. |
|
I don't really believe that screenshot is enough ... V. |
|
Hi gozora, |
|
@manums1983 I'm sorry, but without proper logs I can't help you. V. |
Relax-and-Recover (ReaR) Issue Template
SUSE 12.2 ReaR recovery stuck at system layout restoration
Fill in the following items before submitting a new issue
(quick response is not guaranteed with free support):
SUSE 12.2 (SP2) SAP Server
System is not booting up to get local.conf file
Inte 64 bit
UEFI
The resrore is stuck at "system layout restoration"
The text was updated successfully, but these errors were encountered: