Copy backup restore log into recreated system (related to pull request 1797) #1803

jsmeix · 2018-05-08T13:51:08Z

Type: Enhancement
Impact: Normal
Reference to related issue (URL):
Do not print each files restores by TSM in main output #1797
How was this pull request tested?
Works for me with BACKUP=NETFS.
Not tested BACKUP=TSM but also changed for that
@schabrolles please have a look regarding TSM
in particular related to Use Print outside for loop instead of Log (readability) #1799
Brief description of the changes in this pull request:
Have backup restore log in the new separated directory $VAR_DIR/restore/
and copy that directory contents into the recreated system a the end of
"rear recover" via usr/share/rear/wrapup/default/990_copy_logfile.sh

…t 1797)

jsmeix · 2018-05-08T13:56:17Z

FYI:
This is not yet finished.
Currently it is my very first steps so that you can see early what I do
and provide early feedback if I do something plain wrong.

schabrolles

Remove "/" from $fileplace when creating backup_restore_log_file variable

schabrolles · 2018-05-09T08:40:48Z

usr/share/rear/restore/TSM/default/400_restore_with_tsm.sh

 for num in $TSM_RESTORE_FILESPACE_NUMS ; do
    filespace="${TSM_FILESPACES[$num]}"
+    LogUserOutput "Restoring TSM filespace $filespace"
+    # Create backup restore log file name (a different one for each filespace):
+    backup_restore_log_file=$VAR_DIR/restore/$backup_restore_log_file_prefix-$filespace-restore.log


The issue here is that $fileplace are PATH like "/ /home/ /var/lib/". Then your variable looks like:
backup_restore_log_file=TSM-/var/lib/-restore.log. We have to substitute "/" by something like "_" to avoid issues.

I propose something like:

backup_restore_log_file=$VAR_DIR/restore/$backup_restore_log_file_prefix-ROOT${filespace//\//_}-restore.log

This will generate backup_restore_log_file=TSM-ROOT_var_lib_-restore.log

Many thanks for the information!
I will improve it to get a readable file name.

…uest 1803)

jsmeix · 2018-05-09T16:38:25Z

Should be better now but still not finished.

@schabrolles
see my current usr/share/rear/restore/NETFS/default/400_restore_backup.sh
where now both stdout and stderr are redirected into the backup restore log file,
see the comments in the code for the reason why.

Perhaps also for usr/share/rear/restore/TSM/default/400_restore_with_tsm.sh
both stdout and stderr should be redirected into the backup restore log file
if you like it this way?

schabrolles

The log copy is currently blocked when the restore/layout/recovery sub-directories are already created. The use of mkdir -p should easily solve the problem.

2018-05-10 10:42:25.418151042 Running exit tasks
2018-05-10 10:42:25.419730157 Exit task 'mkdir -p -m 0700 /mnt/local//var/log/rear/recover && cp -p /var/log/rear/rear-rear-sles12-143.log /mnt/local//var/log/rear/recover/rear-rear-sles12-143.log || true'
2018-05-10 10:42:25.423562735 Exit task 'mkdir /mnt/local//var/log/rear/recover/layout && cp -pr /var/lib/rear/layout/* /mnt/local//var/log/rear/recover/layout || true'
mkdir: cannot create directory '/mnt/local//var/log/rear/recover/layout': File exists
2018-05-10 10:42:25.425868407 Exit task 'mkdir /mnt/local//var/log/rear/recover/recovery && cp -pr /var/lib/rear/recovery/* /mnt/local//var/log/rear/recover/recovery || true'
mkdir: cannot create directory '/mnt/local//var/log/rear/recover/recovery': File exists
2018-05-10 10:42:25.428149059 Exit task 'mkdir /mnt/local//var/log/rear/recover/restore && cp -pr /var/lib/rear/restore/* /mnt/local//var/log/rear/recover/restore || true'
mkdir: cannot create directory '/mnt/local//var/log/rear/recover/restore': File exists

schabrolles · 2018-05-10T10:58:42Z

usr/share/rear/wrapup/default/990_copy_logfile.sh

+# (see AddExitTask in _input-output-functions.sh) the ordering of how AddExitTask is called
+# must begin with the to-be-last-run exit task and end with the to-be-first-run exit task:
+if test "$( echo $VAR_DIR/restore/* )" ; then
+    copy_restore_log_exit_task="mkdir $recovery_system_recover_log_dir/restore && cp -pr $VAR_DIR/restore/* $recovery_system_recover_log_dir/restore || true"


I suggest to use mkdir -p this will avoid to black the copy if the directory is already created.

schabrolles · 2018-05-10T10:59:12Z

usr/share/rear/wrapup/default/990_copy_logfile.sh

+    copy_restore_log_exit_task="mkdir $recovery_system_recover_log_dir/restore && cp -pr $VAR_DIR/restore/* $recovery_system_recover_log_dir/restore || true"
+    AddExitTask "$copy_restore_log_exit_task"
+fi
+
 # Do not copy layout and recovery related files for the 'restoreonly' workflow:
 if ! test $WORKFLOW = "restoreonly" ; then
    copy_layout_files_exit_task="mkdir $recovery_system_recover_log_dir/layout && cp -pr $VAR_DIR/layout/* $recovery_system_recover_log_dir/layout || true"


same here: mkdir -p

schabrolles · 2018-05-10T10:59:26Z

usr/share/rear/wrapup/default/990_copy_logfile.sh

+    copy_restore_log_exit_task="mkdir $recovery_system_recover_log_dir/restore && cp -pr $VAR_DIR/restore/* $recovery_system_recover_log_dir/restore || true"
+    AddExitTask "$copy_restore_log_exit_task"
+fi
+
 # Do not copy layout and recovery related files for the 'restoreonly' workflow:
 if ! test $WORKFLOW = "restoreonly" ; then
    copy_layout_files_exit_task="mkdir $recovery_system_recover_log_dir/layout && cp -pr $VAR_DIR/layout/* $recovery_system_recover_log_dir/layout || true"
    copy_recovery_files_exit_task="mkdir $recovery_system_recover_log_dir/recovery && cp -pr $VAR_DIR/recovery/* $recovery_system_recover_log_dir/recovery || true"


same here: mkdir -p

schabrolles · 2018-05-10T11:08:54Z

@jsmeix
I agree with #1803 (comment). We should also redirect stderr to the log file.

schabrolles

To avoid merge conflict, could you also redirect to current TSM log into the $backup_restore_log_file by using -errorlognam=\" $backup_restore_log_file\" dsmc parameter.

schabrolles · 2018-05-10T11:51:43Z

usr/share/rear/restore/TSM/default/400_restore_with_tsm.sh

    # Make sure filespace has a trailing / (for dsmc):
    test "${filespace:0-1}" == "/" || filespace="$filespace/"
-    LogUserOutput "Restoring TSM filespace $filespace"
    Log "Running 'LC_ALL=$LANG_RECOVER dsmc restore $filespace $TARGET_FS_ROOT/$filespace -subdir=yes -replace=all -tapeprompt=no ${TSM_DSMC_RESTORE_OPTIONS[@]}'"


Could you please add the following options to TSM restore: -errorlogname=\"$backup_restore_log_file\". This will also put all the errors reported by TSM into the current log files.

schabrolles · 2018-05-10T11:52:36Z

usr/share/rear/restore/TSM/default/400_restore_with_tsm.sh

    Log "Running 'LC_ALL=$LANG_RECOVER dsmc restore $filespace $TARGET_FS_ROOT/$filespace -subdir=yes -replace=all -tapeprompt=no ${TSM_DSMC_RESTORE_OPTIONS[@]}'"
    # Regarding usage of '0<&6 1>&7 2>&8' see "What to do with stdin, stdout, and stderr" in https://github.com/rear/rear/wiki/Coding-Style
-    LC_ALL=$LANG_RECOVER dsmc restore "$filespace" "$TARGET_FS_ROOT/$filespace" -subdir=yes -replace=all -tapeprompt=no "${TSM_DSMC_RESTORE_OPTIONS[@]}" 0<&6 1>"$TMP_DIR/TSM-restore.log" 2>&8
+    LC_ALL=$LANG_RECOVER dsmc restore "$filespace" "$TARGET_FS_ROOT/$filespace" -subdir=yes -replace=all -tapeprompt=no "${TSM_DSMC_RESTORE_OPTIONS[@]}" 0<&6 1>"$backup_restore_log_file" 2>&8


Same here: add the following options to TSM restore: -errorlogname=\"$backup_restore_log_file\"

@schabrolles
I have a dim feeling that special care is needed when two or more processes "just write"
into the same file because I think I remember that in this case the processes must
not "just write " but "append " into the same file because otherwise the processes
may overwrite each others outputs (at least if output happens simlutaneously).
Currently I have no time to dig into that issue but hopefully later this week
or perhaps next week...

Cf. my comments in my current
https://github.com/jsmeix/rear/blob/ef20c659cfd57c6ef590e96092ff6828d817b7c8/usr/share/rear/restore/NETFS/default/400_restore_backup.sh
currently starting at line 135 that reads

# To be more on the safe side append to the log file '>>' instead of plain writing to it '>' # because when a program (bash in this case) is plain writing to the log file it can overwrite # output of a possibly simultaneously running process that likes to append to the log file # (e.g. when background processes run that also uses the log file for logging) # cf. https://github.com/rear/rear/issues/885#issuecomment-310308763 . . . ... 1>>$backup_restore_log_file 2>&1

I wonder what exactly -errorlogname=$backup_restore_log_file does:
Does it plain write to the file like 2>$backup_restore_log_file does it
(i.e. with truncating existing file content when opening that file)
or does it append to the file like 2>>$backup_restore_log_file
which would be o.k.?

@jsmeix, I just made a test and it works perfectly. I can see the errors sent by TSM multiplexed in the logfile.

…y exists (pull request 1803)

… for TSM (pull request 1803)

jsmeix · 2018-05-17T12:45:22Z

@schabrolles
if you have time for it I would appreciate it if you could test
my latest enhancements regarding TSM backup restore.

jsmeix · 2018-05-23T12:03:21Z

@schabrolles @gdha
feel free to postpone this one to ReaR 2.5
or "just merge" it into ReaR 2.4 if you like it as is.

I think currently not all relevant places are fixed but
I think the current state may be already sufficient
for 'tar' and TSM backup.

gdha · 2018-05-28T08:50:13Z

@schabrolles You may decide to merge the PR for 2.4 or not.

schabrolles · 2018-05-28T09:01:51Z

@gdha,
I will start a test tonight and come back to you tomorrow.

schabrolles

Ok for me... tested with rhel6 rhel7 sles11 sles12 ubuntu and NETFS backup (tar)
And sles12sp2 + TSM.

I think we can merge this one for 2.4

gdha · 2018-05-29T07:46:25Z

@schabrolles Thank you for your intensive testing!

Copy backup restore log into recreated system (related to pull reques…

6660488

…t 1797)

jsmeix added the enhancement Adaptions and new features label May 8, 2018

jsmeix added this to the ReaR v2.4 milestone May 8, 2018

jsmeix self-assigned this May 8, 2018

jsmeix requested a review from schabrolles May 8, 2018 13:51

jsmeix mentioned this pull request May 8, 2018

Use Print outside for loop instead of Log (readability) #1799

Merged

schabrolles requested changes May 9, 2018

View reviewed changes

Enhanced how to create backup restore log for NETFS and TSM (pull req…

0a4a636

…uest 1803)

schabrolles requested changes May 10, 2018

View reviewed changes

gdha approved these changes May 11, 2018

View reviewed changes

jsmeix added 2 commits May 14, 2018 12:17

Using 'mkdir -p' because that causes no error if the directory alread…

ef20c65

…y exists (pull request 1803)

Redirect 'dsmc restore' stdout and stderr into the backup restore log…

6180dcd

… for TSM (pull request 1803)

schabrolles approved these changes May 28, 2018

View reviewed changes

gdha merged commit f914bd1 into rear:master May 29, 2018

jsmeix deleted the copy_backup_restore_log_into_recreated_system_related_to_pull_request_1797 branch June 4, 2018 11:50

jsmeix added the fixed / solved / done label Jun 4, 2018

jsmeix mentioned this pull request Oct 25, 2019

Fishy NETFS 500_make_backup.sh and 400_restore_backup.sh #2265

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Copy backup restore log into recreated system (related to pull request 1797) #1803

Copy backup restore log into recreated system (related to pull request 1797) #1803

jsmeix commented May 8, 2018 •

edited

jsmeix commented May 8, 2018

schabrolles left a comment

schabrolles May 9, 2018

jsmeix May 9, 2018

jsmeix commented May 9, 2018

schabrolles left a comment

schabrolles May 10, 2018

schabrolles May 10, 2018

schabrolles May 10, 2018

schabrolles commented May 10, 2018

schabrolles left a comment

schabrolles May 10, 2018

schabrolles May 10, 2018

jsmeix May 14, 2018 •

edited

schabrolles May 14, 2018

jsmeix commented May 17, 2018

jsmeix commented May 23, 2018

gdha commented May 28, 2018

schabrolles commented May 28, 2018

schabrolles left a comment •

edited

gdha commented May 29, 2018

Copy backup restore log into recreated system (related to pull request 1797) #1803

Copy backup restore log into recreated system (related to pull request 1797) #1803

Conversation

jsmeix commented May 8, 2018 • edited

jsmeix commented May 8, 2018

schabrolles left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsmeix commented May 9, 2018

schabrolles left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

schabrolles commented May 10, 2018

schabrolles left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsmeix May 14, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsmeix commented May 17, 2018

jsmeix commented May 23, 2018

gdha commented May 28, 2018

schabrolles commented May 28, 2018

schabrolles left a comment • edited

Choose a reason for hiding this comment

gdha commented May 29, 2018

jsmeix commented May 8, 2018 •

edited

jsmeix May 14, 2018 •

edited

schabrolles left a comment •

edited