New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot create ISO image >4GB on RHEL6 #581

Closed
wwoo1 opened this Issue Apr 24, 2015 · 6 comments

Comments

Projects
None yet
5 participants
@wwoo1

wwoo1 commented Apr 24, 2015

Hello... I could not backup to ISO images >4GB. I set ISO_MAX_SIZE=10000 and that did not help either. Thanks.

[root@teak rear]# rear -v mkbackup
Relax-and-Recover 1.17.0 / Git
Using log file: /var/log/rear/rear-teak.log
Creating disk layout
Creating root filesystem layout
TIP: To login as root via ssh you need to set up /root/.ssh/authorized_keys or SSH_ROOT_PASSWORD in your configuration file
Copying files and directories
Copying binaries and libraries
Copying kernel modules
Creating initramfs
Encrypting disabled
Creating tar archive '/backups/rear/rear.RPwWr8Nk1dERlAI/tmp/isofs/backup/backup.tar.gz'
Archived 5533 MiB [avg 6112 KiB/sec]OK
Archived 5533 MiB in 928 seconds [avg 6106 KiB/sec]
Preparing 2 ISO images ...
Making additionnal ISO image : rear-teak_01.iso
Wrote ISO image: /backups/rear/rear-teak_01.iso (1.2G)
Making ISO image
ERROR: Could not create ISO image (with /usr/bin/mkisofs)
Aborting due to an error, check /var/log/rear/rear-teak.log for details
Terminated
[root@teak rear]# 

From /var/log/rear/rear-teak.log

2015-04-24 12:34:54 Wrote ISO image: /backups/rear/rear-teak_01.iso (1.2G)
2015-04-24 12:34:54 Including output/ISO/Linux-i386/82_create_iso_image.sh
2015-04-24 12:34:54 Starting '/usr/bin/mkisofs'
2015-04-24 12:34:54 Making ISO image
genisoimage 1.1.9 (Linux)
Scanning .
Scanning ./backup
File ./backup/backup.tar.gz.00 is larger than 4GiB-1.
-allow-limited-size was not specified. There is no way do represent this file size. Aborting.
2015-04-24 12:34:54 ERROR: Could not create ISO image (with /usr/bin/mkisofs)
=== Stack trace ===
Trace 0: /usr/sbin/rear:251 main
Trace 1: /usr/share/rear/lib/mkbackup-workflow.sh:22 WORKFLOW_mkbackup
Trace 2: /usr/share/rear/lib/framework-functions.sh:70 SourceStage
Trace 3: /usr/share/rear/lib/framework-functions.sh:31 Source
Trace 4: /usr/share/rear/output/ISO/Linux-i386/82_create_iso_image.sh:17 source
Trace 5: /usr/share/rear/lib/_input-output-functions.sh:132 StopIfError
Message: Could not create ISO image (with /usr/bin/mkisofs)
===================
2015-04-24 12:34:54 Running exit tasks.
rmdir: removing directory, `/backups/rear/rear.RPwWr8Nk1dERlAI/outputfs'
2015-04-24 12:34:54 Finished in 965 seconds
2015-04-24 12:34:54 Removing build area /backups/rear/rear.RPwWr8Nk1dERlAI
rmdir: removing directory, `/backups/rear/rear.RPwWr8Nk1dERlAI/outputfs'
rmdir: failed to remove `/backups/rear/rear.RPwWr8Nk1dERlAI/outputfs': No such file or directory
rmdir: removing directory, `/backups/rear/rear.RPwWr8Nk1dERlAI'
2015-04-24 12:34:55 End of program reached

@gdha gdha self-assigned this Apr 26, 2015

@gdha

This comment has been minimized.

Show comment
Hide comment
@gdha

gdha Apr 26, 2015

Member

See http://stackoverflow.com/questions/20209263/genisoimage-script-cannot-burn-bigger-than-4gb-but-less-than-4-5gb-file
It was never the purpose of rear to create big ISO images, but with the above link you could tweak rear to make your problem go away. Keep in touch on your progress...

Member

gdha commented Apr 26, 2015

See http://stackoverflow.com/questions/20209263/genisoimage-script-cannot-burn-bigger-than-4gb-but-less-than-4-5gb-file
It was never the purpose of rear to create big ISO images, but with the above link you could tweak rear to make your problem go away. Keep in touch on your progress...

@wwoo1

This comment has been minimized.

Show comment
Hide comment
@wwoo1

wwoo1 Apr 27, 2015

I added -allow-limited-size to $ISO_MKISOFS_BIN in /usr/share/rear/output/ISO/Linux-i386/81_prepare_multiple_iso.sh and the image size was able to go past 4GB, but it looks like 8.6GB is the limit. I worked around it by tar'ing the large directories and manually restoring them after recovery. Thanks for your help.

wwoo1 commented Apr 27, 2015

I added -allow-limited-size to $ISO_MKISOFS_BIN in /usr/share/rear/output/ISO/Linux-i386/81_prepare_multiple_iso.sh and the image size was able to go past 4GB, but it looks like 8.6GB is the limit. I worked around it by tar'ing the large directories and manually restoring them after recovery. Thanks for your help.

@gdha gdha added the enhancement label May 7, 2015

@gdha gdha added this to the Rear v1.18 milestone May 13, 2015

@multipathmaster

This comment has been minimized.

Show comment
Hide comment
@multipathmaster

multipathmaster Sep 1, 2015

So yes, having -allow-limited-size to $ISO_MKISOFS_BIN will enable the ISO image to become larger than 4.7 GIGs. However the real issue is that in RHEL6/CENTOS6 mkisofs is actually a soft link back to /etc/alternatives that points to genisoimage. The issue is that genisomage is not infact mkisofs, with the REAL version of mkisofs being a subset of a tools package called CDRTOOLS. Which is open source, however because of certain limitations of working for a large corporation, we do not have the ability to install (./configure make make install) for the real binary version of mkisofs, so instead, you can always add a new variable within the /usr/share/rear/backup/NETFS/default/50* called SPLIT_COMMAND2 which references the actual split differently than the normal $SPLIT_COMMAND that is called under the tar function sequence of that script.

so inside 50_make_backup.sh you can put this at the top, or inside the set_tar_features

SPLIT_COMMAND2="split --bytes=${BKUP_MAX_SIZE} - ${backuparchive}."

Or name it whatever you wish...

${BKUP_MAX_SIZE} is a variable that you set in /etc/rear/*.conf (whichever conf file you wish to use, os.conf/site.conf/default.conf, whatever you currently are running with rear).

cat /etc/rear/site.conf | grep BKUP_MAX_SIZE
BKUP_MAX_SIZE=3500MB

Now under the tar section of ${BACKUP_PROG} section of 50_make_backup.sh

# tar compatible programs here
        (tar)
                set_tar_features
                Log $BACKUP_PROG $TAR_OPTIONS --sparse --block-number --totals --verbose \
                        --no-wildcards-match-slash --one-file-system \
                        --ignore-failed-read $BACKUP_PROG_OPTIONS \
                        $BACKUP_PROG_X_OPTIONS \
                        ${BACKUP_PROG_BLOCKS:+-b $BACKUP_PROG_BLOCKS} $BACKUP_PROG_COMPRESS_OPTIONS \
                        -X $TMP_DIR/backup-exclude.txt -C / -c -f - \
                        $(cat $TMP_DIR/backup-include.txt) $LOGFILE \| $BACKUP_PROG_CRYPT_OPTIONS BACKUP_PROG_CRYPT_KEY \| $SPLIT_COMMAND2
                $BACKUP_PROG $TAR_OPTIONS --sparse --block-number --totals --verbose \
                        --no-wildcards-match-slash --one-file-system \
                        --ignore-failed-read $BACKUP_PROG_OPTIONS \
                        $BACKUP_PROG_X_OPTIONS \
                        ${BACKUP_PROG_BLOCKS:+-b $BACKUP_PROG_BLOCKS} $BACKUP_PROG_COMPRESS_OPTIONS \
                        -X $TMP_DIR/backup-exclude.txt -C / -c -f - \
                        $(cat $TMP_DIR/backup-include.txt) $LOGFILE | $BACKUP_PROG_CRYPT_OPTIONS $BACKUP_PROG_CRYPT_KEY | $SPLIT_COMMAND2

see how now I am calling on $SPLIT_COMMAND2 at the tail end of these 2 lines?

What does this do for RHEL6?

Your backup.tar.gz will go into your tmp directory (you can define this as well, or leave it default if you think you have enough room), and it will only allow the backup.tar.gz do grow to 3.5 GIGS, and then it splits it off and creates a new one, and you wind up with a naming convention such as this --> backup.tar.gz.aa
backup.tar.gz.ab
backup.tar.gz.ac
backup.tar.gz.ad

so on and so forth. This will keep the backup.tar.gz from exceeding the 4GiB hard limit that is set by the false mkisofs that is really a pointer back to genisoimage. Then once the ISO is created, it can grow to as big as it needs, while copying the backup.tar.gz files within the ISO.

Once it is time for recovery, the recovery section utilizes a for i in ls ${BUILDDIR}/path/to/backups/*.tar.gz which finds the path to those backup.tar.gz and because of globbing within a native shell it should extract them in the same order as it had created them. Although some testing needs to be done.

This simple step will allow much larger than 4 gig ISO images, but keeps the actual NETFS tar.gz files smaller than 4GIB so that you avoid that pesky error about the file size being too large.

multipathmaster commented Sep 1, 2015

So yes, having -allow-limited-size to $ISO_MKISOFS_BIN will enable the ISO image to become larger than 4.7 GIGs. However the real issue is that in RHEL6/CENTOS6 mkisofs is actually a soft link back to /etc/alternatives that points to genisoimage. The issue is that genisomage is not infact mkisofs, with the REAL version of mkisofs being a subset of a tools package called CDRTOOLS. Which is open source, however because of certain limitations of working for a large corporation, we do not have the ability to install (./configure make make install) for the real binary version of mkisofs, so instead, you can always add a new variable within the /usr/share/rear/backup/NETFS/default/50* called SPLIT_COMMAND2 which references the actual split differently than the normal $SPLIT_COMMAND that is called under the tar function sequence of that script.

so inside 50_make_backup.sh you can put this at the top, or inside the set_tar_features

SPLIT_COMMAND2="split --bytes=${BKUP_MAX_SIZE} - ${backuparchive}."

Or name it whatever you wish...

${BKUP_MAX_SIZE} is a variable that you set in /etc/rear/*.conf (whichever conf file you wish to use, os.conf/site.conf/default.conf, whatever you currently are running with rear).

cat /etc/rear/site.conf | grep BKUP_MAX_SIZE
BKUP_MAX_SIZE=3500MB

Now under the tar section of ${BACKUP_PROG} section of 50_make_backup.sh

# tar compatible programs here
        (tar)
                set_tar_features
                Log $BACKUP_PROG $TAR_OPTIONS --sparse --block-number --totals --verbose \
                        --no-wildcards-match-slash --one-file-system \
                        --ignore-failed-read $BACKUP_PROG_OPTIONS \
                        $BACKUP_PROG_X_OPTIONS \
                        ${BACKUP_PROG_BLOCKS:+-b $BACKUP_PROG_BLOCKS} $BACKUP_PROG_COMPRESS_OPTIONS \
                        -X $TMP_DIR/backup-exclude.txt -C / -c -f - \
                        $(cat $TMP_DIR/backup-include.txt) $LOGFILE \| $BACKUP_PROG_CRYPT_OPTIONS BACKUP_PROG_CRYPT_KEY \| $SPLIT_COMMAND2
                $BACKUP_PROG $TAR_OPTIONS --sparse --block-number --totals --verbose \
                        --no-wildcards-match-slash --one-file-system \
                        --ignore-failed-read $BACKUP_PROG_OPTIONS \
                        $BACKUP_PROG_X_OPTIONS \
                        ${BACKUP_PROG_BLOCKS:+-b $BACKUP_PROG_BLOCKS} $BACKUP_PROG_COMPRESS_OPTIONS \
                        -X $TMP_DIR/backup-exclude.txt -C / -c -f - \
                        $(cat $TMP_DIR/backup-include.txt) $LOGFILE | $BACKUP_PROG_CRYPT_OPTIONS $BACKUP_PROG_CRYPT_KEY | $SPLIT_COMMAND2

see how now I am calling on $SPLIT_COMMAND2 at the tail end of these 2 lines?

What does this do for RHEL6?

Your backup.tar.gz will go into your tmp directory (you can define this as well, or leave it default if you think you have enough room), and it will only allow the backup.tar.gz do grow to 3.5 GIGS, and then it splits it off and creates a new one, and you wind up with a naming convention such as this --> backup.tar.gz.aa
backup.tar.gz.ab
backup.tar.gz.ac
backup.tar.gz.ad

so on and so forth. This will keep the backup.tar.gz from exceeding the 4GiB hard limit that is set by the false mkisofs that is really a pointer back to genisoimage. Then once the ISO is created, it can grow to as big as it needs, while copying the backup.tar.gz files within the ISO.

Once it is time for recovery, the recovery section utilizes a for i in ls ${BUILDDIR}/path/to/backups/*.tar.gz which finds the path to those backup.tar.gz and because of globbing within a native shell it should extract them in the same order as it had created them. Although some testing needs to be done.

This simple step will allow much larger than 4 gig ISO images, but keeps the actual NETFS tar.gz files smaller than 4GIB so that you avoid that pesky error about the file size being too large.

@gdha gdha added documentation and removed enhancement labels Nov 1, 2015

@gdha

This comment has been minimized.

Show comment
Hide comment
@gdha

gdha Feb 19, 2016

Member

Added a note to the FAQ of rear (on the web pages)

Member

gdha commented Feb 19, 2016

Added a note to the FAQ of rear (on the web pages)

@Br0adax3

This comment has been minimized.

Show comment
Hide comment
@Br0adax3

Br0adax3 Oct 5, 2016

Uhm. I have run into this problem with an installation on RHEL7.2, using the version that comes with the OS (1.17.2). I have modified the scripts as suggested here, and still got the error. The problem file in my case is ./isolinux/initrd.cgz, which has grown to some 5.8G. Any suggestions?

Thanks.

Br0adax3 commented Oct 5, 2016

Uhm. I have run into this problem with an installation on RHEL7.2, using the version that comes with the OS (1.17.2). I have modified the scripts as suggested here, and still got the error. The problem file in my case is ./isolinux/initrd.cgz, which has grown to some 5.8G. Any suggestions?

Thanks.

@jsmeix

This comment has been minimized.

Show comment
Hide comment
@jsmeix

jsmeix Oct 6, 2016

Contributor

I have a general question:

Currently I cannot imagine a good reason why
one would like to have the backup.tar.gz within the ISO?

"BACKUP_URL=iso://" is mentioned in the documentation
but without a reason when this could be useful or to what
extent this is meant to be used.

What is the advantage to mix up the backup.tar.gz
with the rear recovery system ISO image?

Why not keep separated things separated?

Furthermore in general regarding huge backup.tar.gz see
#1006 (comment)

Finally in general regarding issues with the backup:

Relax-and-Recover (rear) is neither a backup software nor a
backup management software and it is not meant to be one, cf.
"Relax-and-Recover (rear) versus backup and restore" in
https://en.opensuse.org/SDB:Disaster_Recovery

In particular in this case here rear only calls an external tool
that makes an ISO image in the usual way as it is normally
intended for rear i.e. an ISO image that contains only
the rear recovery system (up to about 500 MB)
or at most an ISO image that contains the rear recovery
system plus a backup of the files of the basic operating
system (up to very few GB - usually less than 4 GB)
but not to also a backup of huge application files
like databases and things like that.

Contributor

jsmeix commented Oct 6, 2016

I have a general question:

Currently I cannot imagine a good reason why
one would like to have the backup.tar.gz within the ISO?

"BACKUP_URL=iso://" is mentioned in the documentation
but without a reason when this could be useful or to what
extent this is meant to be used.

What is the advantage to mix up the backup.tar.gz
with the rear recovery system ISO image?

Why not keep separated things separated?

Furthermore in general regarding huge backup.tar.gz see
#1006 (comment)

Finally in general regarding issues with the backup:

Relax-and-Recover (rear) is neither a backup software nor a
backup management software and it is not meant to be one, cf.
"Relax-and-Recover (rear) versus backup and restore" in
https://en.opensuse.org/SDB:Disaster_Recovery

In particular in this case here rear only calls an external tool
that makes an ISO image in the usual way as it is normally
intended for rear i.e. an ISO image that contains only
the rear recovery system (up to about 500 MB)
or at most an ISO image that contains the rear recovery
system plus a backup of the files of the basic operating
system (up to very few GB - usually less than 4 GB)
but not to also a backup of huge application files
like databases and things like that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment