New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Recovery from GRUB menu with Borg failed due to full disk in certain VM environment #1994
Comments
@gozora @Signum Also See also in Attachments, as applicable ("rear -D mkrescue/mkbackup/recover" debug log files): and "Debugging issues with Relax-and-Recover" in |
@jsmeix Just as an enhancement idea: should REAR check if the recovery binary (e.g. "borg") is available and can run before destroying the physical disks and repartitioning them? |
Typo by the way… I meant to say there there is no space left on "/". |
Hello @Signum, Honestly I've never used booting ReaR rescue system from Grub (apart from writing couple of patches...), I simply don't like it ;-).
We are talking here about disaster recovery, your system is already destroyed :-). I'll check what can be done in terms of pre-checks V. |
I happen to have Fedora release 26 (Twenty Six) installed and configured with ReaR Borg back end.
Booted this entry and triggered After booting into ReaR recovery system, I've got following file system layout:
Then after starting
Now I start to think that I did not fully understand what your problem actually is .... How can you have full disk, when running
it should take some time until your disk fills in ... V. |
@Signum I think to analyze what the root cause is here the crucial point is not I think to analyze what the root cause is here the crucial point is |
@gozora I deployed a fresh Debian 9 system from a Vagrant box. I installed the standalone "borg" binary and installed REAR. I have the dim feeling @Signum may try to run 'rear recover' |
@gozora
|
I had overlooked @Signum |
@Signum can you maybe try to add more RAM to your VM? V. |
I tried |
This is my disk usage after booting into the system:
After calling "borg list" it instantly becomes:
…and borg fails with "Failed to write all bytes for libcrypto.so.1.0.0". It looks like BorgBackup has created a directory /tmp/_MEIoKn9nB/ that contains:
…and that's pretty exactly 16 MB in size. So either 210 MB is not enough for the root partition or it is not expected that Borg creates that many files in /tmp. |
@Signum that makes perfect sense. I guess that you just need more RAM assigned to your VM and you are safe ... V. |
@gozora This really seems to be a corner case. It was just unfortunate because REAR destroys the existing partitions and then Borg fails. So perhaps it could be checked if "borg" is executable. But in the end that's really a minor issue. Thanks everyone for lending me your brains. |
@Signum @gozora Currently I don't know what "sufficient free ramdisk space" could be On my KVM/QEMU system with 1GiB memory @Signum I like to get an idea how much free ramdisk space is usually needed. |
@jsmeix question of memory can be quite tricky.
This means that my Arch VM will boot without any trouble with 256MB of RAM.
Which will fail to boot with 256MB of RAM. This is why I think that it might be a bit hard to find universal way how to resolve this ... V. |
I guess @Signum was kind of lucky that he was able to boot, if his initramfs would be ~16MB larger (e.g. by using MODULES=( 'all_modules' )), his ReaR recovery system would not boot at all. V. |
@gozora FYI: # export MIGRATION_MODE=false # rear -D recover & for i in $( seq 120 ) ; do df -h / ; sleep 1 ; done that shows me each second a |
@jsmeix just a short heads up on Arch
Fedora
So yet another complication I guess ... V. |
Heh, even more funny ...
V. |
@gozora |
Only a blind guess: |
Hello @jsmeix, This one looks better:
Excerpt from
V. |
ReaR version ("/usr/sbin/rear -V"): Relax-and-Recover 2.3 / Git
OS version ("cat /etc/rear/os.conf" or "lsb_release -a" or "cat /etc/os-release"): Debian 9
ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat /etc/rear/local.conf"):
OUTPUT=ISO
OUTPUT_URL="nfs://nas/share/mnt/rear"
GRUB_RESCUE=y
SSH_UNPROTECTED_PRIVATE_KEYS=yes
BACKUP=BORG
BORGBACKUP_HOST="nas"
BORGBACKUP_USERNAME="rear"
BORGBACKUP_REPO="/share/rear/ansible-test"
export BORG_PASSPHRASE="foobar"
COPY_AS_IS_BORG=( "/root/.ssh/id_rsa" "/root/rear" )
export BORG_RELOCATED_REPO_ACCESS_IS_OK=yes
export BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=yes
export PRE_BACKUP_SCRIPT=/root/rear/pre_backup
export POST_BACKUP_SCRIPT=/root/rear/post_backup
export POST_RECOVERY_SCRIPT=/root/rear/post_recovery
BACKUP_PROG_INCLUDE=( "${BACKUP_PROG_INCLUDE[@]}" '/dbsnap/*' )
USE_SERIAL_CONSOLE=y
BORGBACKUP_PRUNE_WEEKLY=2
export BACKUP_PROG_EXCLUDE=( "/var/lib/postgresql" )
Hardware (PC or PowerNV BareMetal or ARM) or virtual machine (KVM guest or PoverVM LPAR):
Virtualbox VM with Debian 9 started by Vagrant and deployed by Ansible.
Used for staging of REAR configuration.
AMD64
VirtualBox BIOS
GRUB
Emulated disk via VirtualBox.
I deployed a fresh Debian 9 system from a Vagrant box. I installed the standalone "borg" binary
and installed REAR.
When booting the VM I get "Relax and Recover" as third item in the GRUB boot menu. I can choose that and boot the system. Recovery will start to partition the disks correctly. Then "borg" is called and quits immediately with messages like "Failed to write all bytes for _bisect.so". Apparently that is caused by the root disk being full 100% so borg fails to run.
However when I create a VirtualBox VM manually then there is space left on "/" and recovering from the GRUB menu entry works as expected.
Booting from the ISO image works every time, too.
Boot from ISO.
The text was updated successfully, but these errors were encountered: