Align OUTPUT_URL=null description of man page throughout rear workflows #734

Open
gdha opened this Issue Dec 8, 2015 · 8 comments

Comments

Projects
None yet
2 participants
@gdha
Member

gdha commented Dec 8, 2015

The man page says:

OUTPUT_URL=null
   Do not copy the ISO image from /var/lib/rear/output/ to an external destination.
   Useful in combination with an external backup program, or when BACKUP_URL=iso://backup

but, currently this is only honoured when BACKUP=NETFS method has been set.
We have had requests for external backups methods like TSM and NSR (issues #634, #705, #419 and #501) to reduce duplicate ISOs or to avoid backup ISOs after each rear mkrescue run.

Therefore, we could define some new global variables especially for external backup methods:

  • COPY_LOCAL_ISO_IMAGE= : do not copy (or to copy when 1) the ISO image to external backup media. (default set to 1)
  • REMOVE_LOCAL_ISO_IMAGE= : do not remove (or to remove when 1) ISO image after successful copied to an external backup media. Of course, if we define COPY_LOCAL_ISO_IMAGE=1 then REMOVE_LOCAL_ISO_IMAGE=1 makes sense as the ISO image was successful saved (however, we must be sure it was saved correctly). Default empty.
  • If we define OUTPUT_URL=null we could automatically set COPY_LOCAL_ISO_IMAGE= (do not copy the ISO image to external backup media). However, we would prefer also to see REMOVE_LOCAL_ISO_IMAGE= with this settings then.

Last update: 16 Dec 2016 - changed the variable names to understand better what we are talking about

@gdha gdha self-assigned this Dec 8, 2015

@gdha gdha added this to the Rear v1.18 milestone Dec 8, 2015

@gdha gdha modified the milestones: Rear v1.19, Rear v1.18 Feb 22, 2016

@gdha

This comment has been minimized.

Show comment
Hide comment
@gdha

gdha Feb 22, 2016

Member

Move this to next milestone

Member

gdha commented Feb 22, 2016

Move this to next milestone

@gdha

This comment has been minimized.

Show comment
Hide comment
@gdha

gdha Dec 16, 2016

Member

@jsmeix Does the above make any sense? If not then we can just close this issue

Member

gdha commented Dec 16, 2016

@jsmeix Does the above make any sense? If not then we can just close this issue

@jsmeix

This comment has been minimized.

Show comment
Hide comment
@jsmeix

jsmeix Dec 19, 2016

Contributor

@gdha
in current code I cannot find where OUTPUT_URL=null
is ever used:

$ find usr/sbin/rear usr/share/rear/* | xargs grep -10 'OUTPUT_URL' | grep 'null'
[no output]

Where in current code is OUTPUT_URL=null used?

Contributor

jsmeix commented Dec 19, 2016

@gdha
in current code I cannot find where OUTPUT_URL=null
is ever used:

$ find usr/sbin/rear usr/share/rear/* | xargs grep -10 'OUTPUT_URL' | grep 'null'
[no output]

Where in current code is OUTPUT_URL=null used?

@jsmeix

This comment has been minimized.

Show comment
Hide comment
@jsmeix

jsmeix Dec 19, 2016

Contributor

O.k. - as usual - after asking it - I found
the OUTPUT_URL=null implementation myself:
https://github.com/rear/rear/pull/501/files

Contributor

jsmeix commented Dec 19, 2016

O.k. - as usual - after asking it - I found
the OUTPUT_URL=null implementation myself:
https://github.com/rear/rear/pull/501/files

@jsmeix

This comment has been minimized.

Show comment
Hide comment
@jsmeix

jsmeix Dec 19, 2016

Contributor

For now I will fix the documentation that OUTPUT_URL=null
only works with BACKUP=NETFS.

Contributor

jsmeix commented Dec 19, 2016

For now I will fix the documentation that OUTPUT_URL=null
only works with BACKUP=NETFS.

@jsmeix

This comment has been minimized.

Show comment
Hide comment
@jsmeix

jsmeix Dec 19, 2016

Contributor

@gdha
I fail to understand how OUTPUT_URL=null is useful.
I.e. I do not understand the intent behind
#501 (comment)

With

OUTPUT=ISO
BACKUP=NETFS
BACKUP_OPTIONS="nfsvers=3,nolock"
BACKUP_URL=iso://backup/
OUTPUT_URL=null

I get on the local system

1.3G /tmp/rear.DoVv2d5N5FJMjim/tmp/isofs/backup-basic_system.tar.gz
1.5G var/lib/rear/output/rear-f79.iso

and (as expected) nothing on a NFS output location.

With

OUTPUT=ISO
BACKUP=NETFS
BACKUP_OPTIONS="nfsvers=3,nolock"
BACKUP_URL=iso://backup/
OUTPUT_URL=nfs://10.160.4.244/nfs

I get on the local system

1.3G /tmp/rear.xR2FmUv4D7fEnEf/tmp/isofs/backup-basic_system.tar.gz
1.5G var/lib/rear/output/rear-f79.iso

plus the rear-f79.iso copied to the NFS output location.

Summary from my current experiments:
OUTPUT_URL=null does not save disk space on the local system
but it only prohibits to copy the result to the NFS output location.

I wonder what the benefit of OUTPUT_URL=null is?

Contributor

jsmeix commented Dec 19, 2016

@gdha
I fail to understand how OUTPUT_URL=null is useful.
I.e. I do not understand the intent behind
#501 (comment)

With

OUTPUT=ISO
BACKUP=NETFS
BACKUP_OPTIONS="nfsvers=3,nolock"
BACKUP_URL=iso://backup/
OUTPUT_URL=null

I get on the local system

1.3G /tmp/rear.DoVv2d5N5FJMjim/tmp/isofs/backup-basic_system.tar.gz
1.5G var/lib/rear/output/rear-f79.iso

and (as expected) nothing on a NFS output location.

With

OUTPUT=ISO
BACKUP=NETFS
BACKUP_OPTIONS="nfsvers=3,nolock"
BACKUP_URL=iso://backup/
OUTPUT_URL=nfs://10.160.4.244/nfs

I get on the local system

1.3G /tmp/rear.xR2FmUv4D7fEnEf/tmp/isofs/backup-basic_system.tar.gz
1.5G var/lib/rear/output/rear-f79.iso

plus the rear-f79.iso copied to the NFS output location.

Summary from my current experiments:
OUTPUT_URL=null does not save disk space on the local system
but it only prohibits to copy the result to the NFS output location.

I wonder what the benefit of OUTPUT_URL=null is?

@gdha

This comment has been minimized.

Show comment
Hide comment
@gdha

gdha Dec 19, 2016

Member

@jsmeix Remember, if you do not define an OUTPUT_URL variable it will inherit the value of BACKUP_URL. In case you want the tar archive to be included in the ISO image then we would have 2 big copies of the ISO image (local and remote). The OUTPUT_URL=null prevents the remote copy.
Furthermore, in case a real backup solution back-ups the ISO image to tape (or whatever) then some users want the local copy to be removed as well.
That is the main reason behind this issue.

Member

gdha commented Dec 19, 2016

@jsmeix Remember, if you do not define an OUTPUT_URL variable it will inherit the value of BACKUP_URL. In case you want the tar archive to be included in the ISO image then we would have 2 big copies of the ISO image (local and remote). The OUTPUT_URL=null prevents the remote copy.
Furthermore, in case a real backup solution back-ups the ISO image to tape (or whatever) then some users want the local copy to be removed as well.
That is the main reason behind this issue.

@jsmeix

This comment has been minimized.

Show comment
Hide comment
@jsmeix

jsmeix Dec 20, 2016

Contributor

@gdha
many thanks for your explanation.
Now I understand (at last I hope so ;-)

With
#1132
the issue should be sufficiently documented
to avoid false expectations for ReaR 2.0.

For the future (i.e. for ReaR > 2.0):

In general regarding "remove":

In general I think ReaR should never remove something, cf.
my implementation of BACKUP_RESTORE_MOVE_AWAY
(see my comment in default.conf)

ReaR will not remove any file (any user data is sacrosanct).

In particular regarding REMOVE_LOCAL_ISO_IMAGE:
Instead of first creating a possibly huge local ISO
(when the ISO contains the backup) and then copy
that to the remote output destination, I would prefer to
create the ISO directly at the remote output destination
if possible with the specified OUTPUT_URL scheme.
This would avoid that the local systems disk could get
filled up with the local huge ISO.
In particular when the remote output destination is
a NFS share ReaR could create the ISO directly there
in the same way as it currently writes the backup directly
onto the NFS server with BACKUP_URL=nfs://...

Contributor

jsmeix commented Dec 20, 2016

@gdha
many thanks for your explanation.
Now I understand (at last I hope so ;-)

With
#1132
the issue should be sufficiently documented
to avoid false expectations for ReaR 2.0.

For the future (i.e. for ReaR > 2.0):

In general regarding "remove":

In general I think ReaR should never remove something, cf.
my implementation of BACKUP_RESTORE_MOVE_AWAY
(see my comment in default.conf)

ReaR will not remove any file (any user data is sacrosanct).

In particular regarding REMOVE_LOCAL_ISO_IMAGE:
Instead of first creating a possibly huge local ISO
(when the ISO contains the backup) and then copy
that to the remote output destination, I would prefer to
create the ISO directly at the remote output destination
if possible with the specified OUTPUT_URL scheme.
This would avoid that the local systems disk could get
filled up with the local huge ISO.
In particular when the remote output destination is
a NFS share ReaR could create the ISO directly there
in the same way as it currently writes the backup directly
onto the NFS server with BACKUP_URL=nfs://...

jsmeix added a commit that referenced this issue Dec 20, 2016

Merge pull request #1132 from jsmeix/fix_documentation_regarding_OUTP…
…UT_URL_null_issue734

More explanatory documentation regarding OUTPUT_URL=null
see #734
Additionally more explanatory documentation in default.conf
about proper usage of quoting for BACKUP_PROG_INCLUDE
and BACKUP_PROG_EXCLUDE which is crucial to get
the intended results.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment