Recovery media missing python for borg backup

[catsem]

Hey,
I just tried to recover a system where I used borg as backup system. The recovery fails at the step where rear wants to list the borg repository.

When I try to run borg from shell, it tells me that there is no python interpreter available.
Also made a find / -name 'python*' with no result.

• rear version (/usr/sbin/rear -V): Relax-and-Recover 2.3 / 2017-12-20
• OS version (cat /etc/rear/os.conf or lsb_release -a): Ubuntu 16.04.3 LTS
• rear configuration files (cat /etc/rear/site.conf or cat /etc/rear/local.conf):

OUTPUT=ISO
OUTPUT_URL=file:///tmp/rescue

GRUB_RESCUE=n
ONLY_INCLUDE_VG=( "vg00" )

### Borg stuff ###
BACKUP=BORG
BORGBACKUP_HOST="backup.lan"
BORGBACKUP_USERNAME="rearbackup"
BORGBACKUP_REPO="rearbackup"

# Compression
BORGBACKUP_COMPRESSION="lz4"

# BORG automatic answers env variables
# https://borgbackup.readthedocs.io/en/stable/usage/general.html#environment-variables
export BORG_RELOCATED_REPO_ACCESS_IS_OK="yes"
export BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK="yes"

• Are you using legacy BIOS or UEFI boot? BIOS

I would be glad for any help
Christian

[gozora]

Hi @catsem,

Please check #1030 (comment)

TL;DR
Either you do manual includes of Python, as comment proposes, or use Borg binaries generated with pyinstaller

V.

[catsem]

Hey @gozora,
thanks for your reply.
I think most people with a larger environment will use the python versions of borg, because this is what you get from the package repositorys (at least with Ubuntu/Debian). IMHO rear should take care that we have python in the rescue image out of the box, just like it does for duplicity or yum.
See:
rear/usr/share/rear/prep/DUPLICITY/default/050_prep_duplicity.sh
rear/usr/share/rear/build/DUPLICITY/default/600_create_python_symlink.sh

Could be seen as an enhancement / feature request.

Best
Christian

[gozora]

My knowledge of Python and surrounding infrastructure is very limited and I really don't dare to write code like this.
If you know how to implement this so it works universally across distros , does not break anything, and (last but not least) honors users with Borg binaries generated with pyinstaller (which might not want whole python env copied), you are more then welcome to create PR for this enhancement.

I think most people with a larger environment will use the python versions of borg

I am a "happy" owner or "larger" environment and I personally prefer using Borg binaries generated with pyinstaller ;-).

For now I'm closing this issue as I guess your original question was answered.

V.

[jsmeix]

@gozora
do you think it is possible with reasonable effort
to add a test during "rear mkrescue/mkbackup" that verifies
whether or not Borg programs will work in the recovery system?

I mean during 'build' stage something like build/default/980_verify_rootfs.sh
but as separated script that only runs in case of BACKUP=BORG.

[jsmeix]

@catsem
regarding your #1727 (comment)
"rear should take care that we have python in the rescue image out of the box":

Certainly not "just by default".

Since ReaR exists is does by default take care that its recovery system
is not bigger than needed.
The reason is that the ReaR recovery system is specific for each original system
because it contains specific data of each original system (primarily disk layout data).
This means when ReaR is used for hundreds or thousands of original systems
(which is a main use case of ReaR in business environments)
any "few more megabytes" by default in the recovery system
can make a noticeable difference regarding storage space that is
needed for hundreds or thousands of ReaR recovery systems.
With RECOVERY_UPDATE_URL this can be mitigated, cf.
#841

Regarding "Python in the rescue image":

What would be needed is an automated test to distinguish whether ro not
"Python stuff" is needed to run Borg programs in the recovery system,
and if yes, what exact "Python stuff" is actually needed.

Since ReaR 2.3 we have for usually compiled programs a 'ldd' test
in usr/share/rear/build/default/980_verify_rootfs.sh
but of course - as any automatism it has shortcomings - because
this test cannot autodetect if unsually compiled programs may
require additional special libraries via 'dlopen' at runtime.

I know nothing at all if for Python programs there is a tool
that can autodetect which other "Python stuff" Python programs
may require at runtime.

[gozora]

Hello @jsmeix,

Checking Brog installed from Debian repo I have:

debian:(/home/sodoma/tmp)(sodoma)# ldd $(type -p borg)
	not a dynamic executable

Which is OK, I guess, since we are dealing with Python script here.

Borg binaries generated with pyinstaller on the other hand returns:

debian:(/home/sodoma/tmp)(sodoma)# ldd borg-linux64
	linux-vdso.so.1 (0x00007fffda0fc000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f5541aa6000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f554188c000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f55414ed000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f5541caa000)

So distinguish between Borg script and Borg binaries generated with pyinstaller is rather easy.
As I've pointed out in #1727 (comment) I have zero experience with Python and I don't dare to decide what libs / env / configs must by copied for Borg into ReaR to make it work correctly, neither to write some test code and decide if Borg would be able to recover system successfully. Not talking about fact that every distro might have different locations for Python environment ...

What I would do is update of documentaiton and warn users that until they don't use Borg binaries generated with pyinstaller, they need to include all necessary Python files.

V.

[jsmeix]

@gozora
in #1727 (comment)
I only meant an automated test during 'rear mkrescue'
that warns users if they do not use Borg binaries.

In your #1727 (comment)
I noticed that another positive side-effect of using Borg binaries is
that the generic 'ldd' test in build/default/980_verify_rootfs.sh
checks that the Borg binaries have all their needed libraries
in the recovery system.

[gozora]

I only meant an automated test during 'rear mkrescue'
that warns users if they do not use Borg binaries.

This is something that could be done with "reasonable effort" + documentation update mentioned in #1727 (comment).
But I'd really not try to automatically copy Python env into ReaR recovery system.

V.

[gozora]

Especially not when there is very elegant solution to address this problem, directly from Borg guys ;-) ...

V.

[gozora]

@jsmeix

Do you think we can use code like this near the end of build/default/980_verify_rootfs.sh ?

if [[ $BACKUP = "BORG" ]]; then
	ldd $(type -p borg) > /dev/null
	LogPrintIfError "Make sure you have correctly included Python environment to make Borg work correctly"
fi

V.

[jsmeix]

@gozora
#1727 (comment)
is perfectly fine with me!
Just do a pull request and merge it
(preferably with a comment directly in the code that points to this issue
so that at any time later also others understand the reason behind ;-)

[jsmeix]

Automatically copy "the right needed" Python enviroment into ReaR recovery system
is left as an exercise for the reader... ;-)

Let's see when we need Java support in the ReaR recovery system ;-))

[gozora]

@jsmeix you nailed it with that Java comment! :-)

[gozora]

Documentation updated in #1730.

I'll add Borg binary vs script check / issue warning in separate PR.

V.