Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

precompiled borg binaries on qnap NAS #690

Closed
copart opened this Issue Feb 22, 2016 · 30 comments

Comments

Projects
None yet
10 participants
@copart
Copy link

copart commented Feb 22, 2016

Hi,

I have a qnap NAS (x86_64 arch) that I am trying to get borg working with. At this point I am unable to get multiple python packages to compile (the device uses a toolset from the openwrt project for customer stuff). Hence, trying to precompiled option.

Anyway, I have Attic v0.14 working on the qnap using a precompiled binary. This works perfectly. I attempted to use borg-linux64 and the executable segfaults on run. Should it work?

Anybody else get Python3, numpy, and other packages working on a qnap using Entware-ng for gcc and qpkg for Python 3?

@ThomasWaldmann

This comment has been minimized.

Copy link
Member

ThomasWaldmann commented Feb 22, 2016

Please check what glibc version you have on the NAS device.

If it is much older than the one used to build the binary, the crash is likely due to that (see the docs / binary README).

@copart

This comment has been minimized.

Copy link
Author

copart commented Feb 22, 2016

Thanks for replying. It appears the the Qnap has a recent glibc, at 2.22. Same as my fedora install.

# uname -a
Linux qnap 3.4.6 #1 SMP Tue Aug 4 09:57:04 CST 2015 x86_64 GNU/Linux
# LD_LIBRARY_PATH=/opt/lib ./getglibcver 
2.22
# ldd --version | grep ldd
ldd (GNU libc) 2.22
# LD_LIBRARY_PATH=/opt/lib ./borg-linux64
Segmentation fault
# LD_LIBRARY_PATH=/opt/lib TEMP=/share/dir ./borg-linux64
Segmentation fault

I tried to move where it extracted as another try.

@ThomasWaldmann

This comment has been minimized.

Copy link
Member

ThomasWaldmann commented Feb 22, 2016

Why are you tweaking LD_LIBRARY_PATH?

pyinstaller does also tweak LD_LIBRARY_PATH, so all the bundled libs get loaded from the directory where the stuff gets unpacked.

@copart

This comment has been minimized.

Copy link
Author

copart commented Feb 23, 2016

hi, the Qnap has its own built in libraries that I believe conflict if you do not tweak LD_LIBRARY_PATH. If I do not export LD_LIBRARY_PATH or set on cmdline, it loads the built in and not the ones in the /opt directories.

For example:

TEMP=/share/dir ./borg-linux64

./borg-linux64: error while loading shared libraries: libdl.so.2: wrong ELF class: ELFCLASS32

Just to confirm that 32 bit does not work either as the above kind of hints at architecture mismatch

TEMP=/share/dir ./borg-linux32

Error loading Python lib '/share/dir/_MEIWLgWaV/libpython3.5m.so.1.0': /lib/libc.so.6: version `GLIBC_2.7' not found (required by /share/dir/_MEIWLgWaV/libpython3.5m.so.1.0)

@manwegit

This comment has been minimized.

Copy link

manwegit commented Feb 25, 2016

I'd also like to get borg working on Qnap, but atm I'm running stuff with nfs mounts from Qnap.
Default libs are old and most likely will not be updated for the foreseeable future. Let me know if you can get it to work.

@ThomasWaldmann ThomasWaldmann changed the title Precompiled binaries precompiled binaries on qnap NAS Feb 25, 2016

@ThomasWaldmann ThomasWaldmann changed the title precompiled binaries on qnap NAS precompiled borg binaries on qnap NAS Feb 25, 2016

@manwegit

This comment has been minimized.

Copy link

manwegit commented Mar 4, 2016

@copart Progessing on Qnap build, but I'm missing some dev headers and libs. I'll try to get some progress on this next week.
BUT I'm not sure, will I create standalone bin. Steps to build on Qnap borg is my first priority.

@copart

This comment has been minimized.

Copy link
Author

copart commented Mar 4, 2016

I wish I could figure out how to install Python3 libraries that require compilation. They all seem to fail even with the headers available from Entware-ng downloaded and proper env setup. This way I assume I could get borgbackup installed via pip.

@nepet

This comment has been minimized.

Copy link

nepet commented Mar 12, 2016

Hi there,
have you made any progress on the issue by now?

@copart

This comment has been minimized.

Copy link
Author

copart commented Mar 13, 2016

@nepet
Not yet, however. have not tried in the last week. Please let us know if you find out.

@manwegit

This comment has been minimized.

Copy link

manwegit commented Mar 14, 2016

Sorry, busy. looks like I'll be busy this week also, so nothing new (most likely) this week from me.

@Saw-Head

This comment has been minimized.

Copy link

Saw-Head commented Apr 8, 2016

Hey heros,
I love BorgBackup and I am using it on one of my Linux servers (Debian). Absolutely great!
Now I would like to use it on my (old) QNAP NAS TS-210 too.

I tried the original precompiled binaries, but I didn't succeed (cannot execute binary file), guess this is because of the ARM based hardware/software.
I googled a lot but I couldn't find a package that comes without the need of compilation.

Has anyone of you found a solution for this or can someone give me a hint how I could do this?

Cheers,
Saw

@copart

This comment has been minimized.

Copy link
Author

copart commented May 23, 2016

I got it to work, however, I have a QNAP with an ATOM cpu, so YMMV with ARM. I had to BUILD my own Python since the Python3 offered in qpkg is Python i686 which added complexity with regards to CPYTHON.

  1. Install Entware-ng, I forget all the packages I installed, but you obviously need gcc and more
  2. Download headers, set environment
    https://github.com/Entware-ng/Entware-ng/wiki/Using-gcc-%28native-compilation%29,
  3. Download and install lz4. Make sure headers are in /opt/include and library is in /opt/lib. Get from https://github.com/Cyan4973/lz4
  4. Download and BUILD Python, in order for some of the default modules to install I needed to do
    ln -s /opt/lib/libgdbm.so.4.0.0 /opt/lib/libgdbm.so
    ln -s /opt/lib/libssl.so.1.0.0 /opt/lib/libssl.so
    ln -s /opt/lib/libsqlite3.so.0.8.6 /opt/lib/libsqlite3.so
    ln -s /opt/lib/libbz2.so.1.0.6 /opt/lib/libbz2.so
  5. /opt/bin/pip3.5 install borgbackup
  6. BACKUP

Footnote: I did make some changes to Modules/Setup, however, I think that can be avoided as long as you do the ln -s in step 4.

@copart

This comment has been minimized.

Copy link
Author

copart commented Jul 14, 2016

for completeness, I updated my borg install to 1.0.6. I had to do the following:

export BORG_LZ4_PREFIX=/opt
export BORG_OPENSSL_PREFIX=/opt/include

@ThomasWaldmann

This comment has been minimized.

Copy link
Member

ThomasWaldmann commented Jul 14, 2016

@copart you are sure about /opt/include? what is the full path of openssl/evp.h?

@copart

This comment has been minimized.

Copy link
Author

copart commented Aug 3, 2016

@ThomasWaldmann

[~] # ls -la /opt/include/openssl/evp.h
-rw-r--r-- 1 admin administ 67798 Jun 17 19:55 /opt/include/openssl/evp.h

@ThomasWaldmann

This comment has been minimized.

Copy link
Member

ThomasWaldmann commented Aug 3, 2016

I'ld say the openssl prefix is /opt then.

@ThomasWaldmann

This comment has been minimized.

Copy link
Member

ThomasWaldmann commented Nov 25, 2016

no reply, closing.

@azazel75

This comment has been minimized.

Copy link

azazel75 commented Dec 7, 2016

i think i'll try borg inside a debian/ubuntu docker container... my qnap can run them, it seems much simpler

@demure

This comment has been minimized.

Copy link

demure commented Feb 9, 2017

For future people that try to do this on QNAP, the app "Container Station" is the effective solution, using an OS of your choice. You may however face issues with the port forwarding, and have to dive into the QNAP filesystem to manually fix. QNAP Forum post on fix

I will edit this with a link to my blog post when I'm finished writing it, in case the QNAP forum has issues.
EDIT: my blog post

@quicktrick

This comment has been minimized.

Copy link

quicktrick commented Feb 21, 2018

Hello! A lot of thanks to @copart and @ThomasWaldmann ! I was able to install borgbackup on Qnap TS-459 Pro (Linux TS-459 3.4.6 #1 SMP Fri Dec 8 01:32:36 CST 2017 x86_64 GNU/Linux). I installed Entware-3x, followed the steps in @copart 's instruction except I installed python3 from Entware's repository. But I wasn't able to install FUSE support. Here are my logs:

root@TS-459:/share/MD0_DATA/.qpkg/Entware-3x# pip3.6 install borgbackup
Collecting borgbackup
  Using cached borgbackup-1.1.4.tar.gz
Requirement already satisfied: msgpack-python>=0.4.6 in ./lib/python3.6/site-packages (from borgbackup)
Building wheels for collected packages: borgbackup
  Running setup.py bdist_wheel for borgbackup ... done
  Stored in directory: /opt/root/.cache/pip/wheels/9d/85/82/0b24ab0df228e284a03eb02986cc407de283cd175172175cae
Successfully built borgbackup
Installing collected packages: borgbackup
Successfully installed borgbackup-1.1.4

root@TS-459:/share/MD0_DATA/.qpkg/Entware-3x# pip3.6 install borgbackup[fuse]
Requirement already satisfied: borgbackup[fuse] in ./lib/python3.6/site-packages
Requirement already satisfied: msgpack-python>=0.4.6 in ./lib/python3.6/site-packages (from borgbackup[fuse])
Collecting llfuse<2.0; extra == "fuse" (from borgbackup[fuse])
  Using cached llfuse-1.3.2.tar.bz2
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/share/MD0_DATA/.qpkg/Entware-3x/tmp/pip-build-n4dfhnoy/llfuse/setup.py", line 283, in <module>
        main()
      File "/share/MD0_DATA/.qpkg/Entware-3x/tmp/pip-build-n4dfhnoy/llfuse/setup.py", line 76, in main
        compile_args = pkg_config('fuse', cflags=True, ldflags=False, min_ver='2.8.0')
      File "/share/MD0_DATA/.qpkg/Entware-3x/tmp/pip-build-n4dfhnoy/llfuse/setup.py", line 175, in pkg_config
        if subprocess.call(cmd) != 0:
      File "/subprocess.py", line 267, in call
      File "/subprocess.py", line 707, in __init__
      File "/subprocess.py", line 1333, in _execute_child
    FileNotFoundError: [Errno 2] No such file or directory: 'pkg-config'

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /share/MD0_DATA/.qpkg/Entware-3x/tmp/pip-build-n4dfhnoy/llfuse/

What may be the problem with Fuse?

BTW, I was not able to install borg-env too:

root@TS-459:/share/MD0_DATA/.qpkg/Entware-3x# virtualenv --python=python3 borg-env
Running virtualenv with interpreter /opt/bin/python3
Using base prefix '/opt'
New python executable in /share/MD0_DATA/.qpkg/Entware-3x/borg-env/bin/python3
Also creating executable in /share/MD0_DATA/.qpkg/Entware-3x/borg-env/bin/python
Installing setuptools, pip, wheel...
  Complete output from command /share/MD0_DATA/.qpk...borg-env/bin/python3 - setuptools pip wheel:
  Traceback (most recent call last):
  File "<stdin>", line 7, in <module>
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load
  File "<frozen importlib._bootstrap>", line 950, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 646, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 616, in _load_backward_compatible
  File "/opt/lib/python3.6/site-packages/virtualenv_support/pip-9.0.1-py2.py3-none-any.whl/pip/__init__.py", line 26, in <module>
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load
  File "<frozen importlib._bootstrap>", line 950, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 646, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 616, in _load_backward_compatible
  File "/opt/lib/python3.6/site-packages/virtualenv_support/pip-9.0.1-py2.py3-none-any.whl/pip/utils/__init__.py", line 23, in <module>
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load
  File "<frozen importlib._bootstrap>", line 950, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 646, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 616, in _load_backward_compatible
  File "/opt/lib/python3.6/site-packages/virtualenv_support/pip-9.0.1-py2.py3-none-any.whl/pip/locations.py", line 9, in <module>
  File "/share/MD0_DATA/.qpkg/Entware-3x/borg-env/lib/python3.6/distutils/__init__.py", line 17, in <module>
    real_distutils = imp.load_module("_virtualenv_distutils", None, distutils_path, ('', '', imp.PKG_DIRECTORY))
  File "/imp.py", line 244, in load_module
  File "/imp.py", line 210, in load_package
ValueError: '/opt/lib/python3.6/distutils/__init__.py/__init__.pyc' is not a package
----------------------------------------
...Installing setuptools, pip, wheel...done.
Traceback (most recent call last):
  File "/opt/lib/python3.6/site-packages/virtualenv.py", line 2328, in <module>
    main()
  File "/opt/lib/python3.6/site-packages/virtualenv.py", line 713, in main
    symlink=options.symlink)
  File "/opt/lib/python3.6/site-packages/virtualenv.py", line 945, in create_environment
    download=download,
  File "/opt/lib/python3.6/site-packages/virtualenv.py", line 901, in install_wheel
    call_subprocess(cmd, show_stdout=False, extra_env=env, stdin=SCRIPT)
  File "/opt/lib/python3.6/site-packages/virtualenv.py", line 797, in call_subprocess
    % (cmd_desc, proc.returncode))
OSError: Command /share/MD0_DATA/.qpk...borg-env/bin/python3 - setuptools pip wheel failed with error code 1

Anyway, Borg itself is working fine, many-many thanks for that! Though Fuse is very desirable to get working too.

@ThomasWaldmann

This comment has been minimized.

Copy link
Member

ThomasWaldmann commented Feb 22, 2018

llfuse install: it wants the pkg-config tool. see our install docs for some required packages hints.

not sure about the virtualenv issue, it just looks broken (but that is unrelated to borg).

@quicktrick

This comment has been minimized.

Copy link

quicktrick commented Feb 23, 2018

Thank you, Thomas! Unfortunately, that didn't work too. I tried it on the same and on another Qnap NAS. I tried to install it from the source code too, but got the following output:

pip3.6 install -r requirements.d/fuse.txt
Collecting llfuse<2.0 (from -r requirements.d/fuse.txt (line 3))
  Downloading llfuse-1.3.2.tar.bz2 (348kB)
    100% | 348kB 196kB/s
No files/directories in /share/MD0_DATA/.qpkg/Entware-ng/tmp/pip-build-m1ebksv0/llfuse/pip-egg-info (from PKG-INFO)

And I didn't find some required packages in Entware's repository too.

It's not a big deal though. It's mountable remotely!

@ThomasWaldmann

This comment has been minimized.

Copy link
Member

ThomasWaldmann commented Feb 23, 2018

@quicktrick that looks like building llfuse did not work. Maybe -v helps?

And, as I already said, we've documented all requirements for borg (and deps) in the install docs.

@noplanman

This comment has been minimized.

Copy link

noplanman commented Sep 28, 2018

@copart You may want to try building the binary using my docker building project here:
https://git.feneas.org/noplanman/borg-binary-builder

After installing docker and checking out the project ./build qemux86-64 1.1.7 will create an image with a built binary, and ./extract qemux86-64 borg will then extract it so that you can use it directly.

I'd encourage you to read the readme first though!

@quicktrick

This comment has been minimized.

Copy link

quicktrick commented Jan 30, 2019

I've just successfully upgraded from 1.1.4 to 1.1.8 on my old Qnap TS-239 Pro II+ Turbo (x86_32) Linux QNAP 3.4.6 #1 SMP Fri Dec 8 10:21:53 CST 2017 i686 GNU/Linux.

[admin@QNAP src]# pip3.6 install --upgrade pip
[admin@QNAP src]# curl -O -L https://github.com/borgbackup/borg/releases/download/1.1.8/borgbackup-1.1.8.tar.gz
[admin@QNAP src]# tar -xzf borgbackup-1.1.8.tar.gz
[admin@QNAP borgbackup-1.1.8]# TMPDIR=/opt/tmp pip3.6 install -v -r requirements.d/development.txt
[admin@QNAP borgbackup-1.1.8]# TMPDIR=/opt/tmp pip3.6 install -v -r requirements.d/docs.txt
[admin@QNAP borgbackup-1.1.8]# TMPDIR=/opt/tmp pip3.6 install -e .
[admin@QNAP borgbackup-1.1.8]# borg -V
borg 1.1.8

First I installed the github master branch version borgbackup-1.2.0.dev745+g2bcff382, and it seemed to work, but when I tried to backup I got errors, so I moved to 1.1.8. It works fine!

@ThomasWaldmann

This comment has been minimized.

Copy link
Member

ThomasWaldmann commented Jan 30, 2019

yeah, don't use master branch for production.

either a release or at least stuff from 1.1-maint branch.

@ngondard

This comment has been minimized.

Copy link

ngondard commented Feb 26, 2019

Thanks to @ThomasWaldmann, @copart, @quicktrick and everyone else involved in this thread, I got me borgbackup-1.1.8 on an outdated armv5 Buffalo Linkstation!
For information sake, I took the Entware route as @copart explained (first installing python3 + python3-pip and downloading Entware headers into /opt/include). Then I had trouble with openssl. I compiled it from source with Entware packages (namely perl, perlbase-base, perlbase-cpan, gcc, make).
Backuping just fine right now!

@noplanman

This comment has been minimized.

Copy link

noplanman commented Mar 2, 2019

@ngondard Out of curiosity, did you try out the binary at https://borg.bauerj.eu?

@ngondard

This comment has been minimized.

Copy link

ngondard commented Mar 3, 2019

@noplanman it would have spared me quite some time indeed! However, I am currently building borgbackup on another antique (a powerpc-based Mybooklive). Therefore I wanted to learn (besides, I am an IT guy after all).

@noplanman

This comment has been minimized.

Copy link

noplanman commented Mar 3, 2019

Ok! Just checking to see if it worked, to assure that the builds play nicely on different systems 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.