udevsettle command not found, server hangs

[manums1983]

Relax-and-Recover (ReaR) Issue Template

Fill in the following items before submitting a new issue
(quick response is not guaranteed with free support):

• ReaR version ("/usr/sbin/rear -V"):
  2.3 git

• OS version ("cat /etc/rear/os.conf" or "lsb_release -a" or "cat /etc/os-release"):

NAME="SLES"
VERSION="12-SP2"
VERSION_ID="12.2"
PRETTY_NAME="SUSE Linux Enterprise Server 12 SP2"
ID="sles"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:suse:sles:12:sp2"

• ReaR configuration files ("cat /etc/rear/site.conf" or "cat /etc/rear/local.conf"):

# Begin example setup for SLE12-SP2 with default btrfs subvolumes.
# Since SLE12-SP1 what is mounted at '/' is a btrfs snapshot subvolume
# see https://github.com/rear/rear/issues/556
# and since SLE12-SP2 btrfs quota via "snapper setup-quota" is needed
# see https://github.com/rear/rear/issues/999
# You must adapt "your.NFS.server.IP/path/to/your/rear/backup" at BACKUP_URL.
# You must decide whether or not you want to have /home/* in the backup.
# It depends on the size of your harddisk whether or not /home is by default
# a btrfs subvolume or a separated xfs filesystem on a separated partition.
# You may activate SSH_ROOT_PASSWORD and adapt the "password_on_the_rear_recovery_system".
# For basic information see the SLE12-SP2 manuals.
# Also see the support database article "SDB:Disaster Recovery"
# at http://en.opensuse.org/SDB:Disaster_Recovery
# In particular note:
# There is no such thing as a disaster recovery solution that "just works".
# Regarding btrfs snapshots:
# Recovery of btrfs snapshot subvolumes is not possible.
# Only recovery of "normal" btrfs subvolumes is possible.
# On SLE12-SP1 and SP2 the only exception is the btrfs snapshot subvolume
# that is mounted at '/' but that one is not recreated but instead
# it is created anew from scratch during the recovery installation with the
# default first btrfs snapper snapshot subvolume path "@/.snapshots/1/snapshot"
# by the SUSE tool "installation-helper --step 1" (cf. below).
# Other snapshots like "@/.snapshots/234/snapshot" are not recreated.
# Create rear recovery system as ISO image:
OUTPUT=USB
#OUTPUT=ISO
# Store the backup file via NFS on a NFS server:
BACKUP=NETFS
#BACKUP=DP
# BACKUP_OPTIONS variable contains the NFS mount options and
# with 'mount -o nolock' no rpc.statd (plus rpcbind) are needed:
#BACKUP_OPTIONS="nfsvers=3,nolock"
# If the NFS server is not an IP address but a hostname,
# DNS must work in the rear recovery system when the backup is restored.
BACKUP_URL=usb:///dev/disk/by-label/REAR-000
# Keep an older copy of the backup in a HOSTNAME.old directory
# provided there is no '.lockfile' in the HOSTNAME directory:
NETFS_KEEP_OLD_BACKUP_COPY=yes
# Have all modules of the original system in the recovery system with the
# same module loading ordering as in the original system by using the output of
#   lsmod | tail -n +2 | cut -d ' ' -f 1 | tac | tr -s '[:space:]' ' '
# as value for MODULES_LOAD (cf. https://github.com/rear/rear/issues/626):
#MODULES_LOAD=( )
# On SLE12-SP1 and SP2 with default btrfs subvolumes what is mounted at '/' is a btrfs snapshot subvolume
# that is controlled by snapper so that snapper is needed in the recovery system.
# In SLE12-SP1 and SP2 some btrfs subvolume directories (/var/lib/pgsql /var/lib/libvirt/images /var/lib/mariadb)
# have the "no copy on write (C)" file attribute set so that chattr is required in the recovery system
# and accordingly also lsattr is useful to have in the recovery system (but not strictly required):
REQUIRED_PROGS=( "${REQUIRED_PROGS[@]}" snapper chattr lsattr )
# Snapper setup by the recovery system uses /usr/lib/snapper/installation-helper
# that is linked to all libraries where snapper is linked to
# (except libdbus that is only needed by snapper).
# "installation-helper --step 1" creates a snapper config based on /etc/snapper/config-templates/default
COPY_AS_IS=( "${COPY_AS_IS[@]}" /usr/lib/snapper/installation-helper /etc/snapper/config-templates/default )
# Files in btrfs subvolumes are excluded by 'tar --one-file-system'
# so that such files must be explicitly included to be in the backup.
# Files in the following SLE12-SP2 default btrfs subvolumes are
# in the below example not included to be in the backup
#   /.snapshots  /var/crash
# but files in /home are included to be in the backup.
# You may use a command like
#   findmnt -n -r -o TARGET -t btrfs | grep -v '^/$' | egrep -v 'snapshots|crash'
# to generate the values:
BACKUP_PROG_INCLUDE=( /var/cache /var/lib/mailman /var/tmp /var/lib/pgsql /usr/local /opt /var/lib/libvirt/images /boot/grub2/i386-pc /var/opt /srv /boot/grub2/x86_64-efi /var/lib/mariadb /var/spool /var/lib/mysql /tmp /home /var/log /var/lib/named /var/lib/machines )
BACKUP_PROG_EXCLUDE=( "${BACKUP_PROG_EXCLUDE[@]}" "/var/crash" "/export/Doc" "/export/archive" "/usr/sap/hostctrl" "/home/oracle" "/home/oraprd" "/oracle" "/oracle/PRD" "/oracle/PRD/12102" "/oracle/PRD/mirrlogA"  "/oracle/PRD/oraarch" "/oracle/PRD/origlogB" "/oracle/client" "/oracle/oraprd" "/oracle/stage" "/oracle/PRD/mirrlogB" "/oracle/PRD/origlogA" "/oracle/PRD/sapreorg" "/sapmnt/PRD" "/sapmnt/PRD/exe" "/usr/sap/PRD" "/usr/sap/SMD" "/usr/sap/tmp" "/oracle/PRD/sapdata1" "/oracle/PRD/sapdata2" "/oracle/PRD/sapdata3" "/oracle/PRD/sapdata4" "/oracle/PRD/sapdata5" "/oracle/PRD/sapdata6")
EXCLUDE_RECREATE=( "${EXCLUDE_RECREATE[@]}" "fs:/var/crash" "fs:/export/Doc" "fs:/export/archive" "fs:/usr/sap/hostctrl" "fs:/home/oracle" "fs:/home/oraprd" "fs:/oracle" "fs:/oracle/PRD" "fs:/oracle/PRD/12102" "fs:/oracle/PRD/mirrlogA" "fs:/oracle/PRD/oraarch" "fs:/oracle/PRD/origlogB" "fs:/oracle/client" "fs:/oracle/oraprd" "fs:/oracle/stage" "fs:/oracle/PRD/mirrlogB" "fs:/oracle/PRD/origlogA" "fs:/oracle/PRD/sapreorg" "fs:/sapmnt/PRD" "fs:/sapmnt/PRD/exe" "fs:/usr/sap/PRD" "fs:/usr/sap/SMD" "fs:/usr/sap/tmp" "fs:/oracle/PRD/sapdata1" "fs:/oracle/PRD/sapdata2" "fs:/oracle/PRD/sapdata3" "fs:/oracle/PRD/sapdata4" "fs:/oracle/PRD/sapdata5" "fs:/oracle/PRD/sapdata6" )
# The following POST_RECOVERY_SCRIPT implements during "rear recover"
# btrfs quota setup for snapper if that is used in the original system:
POST_RECOVERY_SCRIPT=( 'if snapper --no-dbus -r $TARGET_FS_ROOT get-config | grep -q "^QGROUP.*[0-9]/[0-9]" ; then snapper --no-dbus -r $TARGET_FS_ROOT set-config QGROUP= ; snapper --no-dbus -r $TARGET_FS_ROOT setup-quota && echo snapper setup-quota done || echo snapper setup-quota failed ; else echo snapper setup-quota not used ; fi' )
# This option defines a root password to allow SSH connection
# whithout a public/private key pair
SSH_ROOT_PASSWORD="password"
# Let the rear recovery system run dhclient to get an IP address
# instead of using the same IP address as the original system:
#USE_DHCLIENT="yes"
# End example setup for SLE12-SP2 with default btrfs subvolumes.

• System architecture (x86 compatible or POWER and/or what kind of virtual machine):
  x86_64

• Are you using BIOS or UEFI or another way to boot?
  UEFI

• Brief description of the issue:
  The server getting hung after it boots from the boot image which created by rear. Noticed it is an udevsettle issue.

Debug log captured:

++ grep -v grep
++ grep -q systemd-udev
++ my_udevtrigger
++type -p udevadm
++ udevadm trigger
+++ ps --no-headers -C systemd
++[[ -n 1? 00:01:23 systemd ]]
++ sleep 1
++ my_udevsettle
++ type -p udevadm
++ udevadm settle -- timeout=10
++ udevsettle
/etc/scripts/system-setup.d/00-fumction.sh: line 16: udevsettle: command not found
++ udevadm trigger --action=add
++ echo-n 'waiting for udev ...'
waiting for udev ... ++ sleep 1
++ my_udevsettle
++ type -p udevadm
++ udevadm settle --timeout=10
++ echo done.
done.
++ modeprobe -q dm-mod

Press ENTER to run 41-load-special-modules.sh

• Work-around, if any: NO

[manums1983]

I have attached a dmesg out put. Appreciate you help in resolving this.
gesprd1_ilo_logs_may2018 - Copy (2).txt

[32m[ 2437.614241] �[0m�[33mhpsa 0000:02:00.0�[0m: logical_reset scsi 0:1:0:0: Direct-Access HP LOGICAL VOLUME RAID-0 SSDSmartPathCap- En- Exp=1 qd=0

[manums1983]

work around:
Set the values "rd.udev.event-timeout=300 verbose showopts" in the Relax and recover setparams. Then the restore is started using the command "rear -d -Dv recover" from a putty session got successful.

The value ( rd.udev.event-timeout=300) given from the below tech note.
https://www.suse.com/support/kb/doc/?id=7022681

would there be any way we can put the value persistent in a rear script to avoid the hang issue during the boot up.

[jsmeix]

I wonder about the

/etc/scripts/system-setup.d/00-fumction.sh: line 16: udevsettle: command not found

message because there is no file in ReaR called 00-fumction.sh.
In ReaR there is only this file:
usr/share/rear/skel/default/etc/scripts/system-setup.d/00-functions.sh

@manums1983
do you shomehow have a file 00-fumction.sh or is that a typo?

Please post your
usr/share/rear/skel/default/etc/scripts/system-setup.d/00-f*
files here.

FYI:
udevsettle is a very old way that is theer only as last fallback
nowadays udevadm settle is used.

[manums1983]

That is a typo error. it is actually "00-functions.sh". please see in the screenshots.

[jsmeix]

You can add arbitrary kernel command line parameters for booting the rescue/recovery system
by setting the KERNEL_CMDLINE config variable, see the documentation in default.conf.