[Request] Add numpy to octoprint virtualenv

[jneilliii]

What were you doing?

Installing the Bed Level Visualizer plugin, and I believe also the Prusa Mesh Map plugin numpy has to download and compile, which takes forever on some older devices as well as smaller memory devices.

What did you expect to happen?

This is expected behavior, but would like to improve the user experience.

What happened instead?

The plugin took forever to install.

Did the same happen when running OctoPrint in safe mode?

N/a

Version of OctoPi

0.17

Printer model & used firmware incl. version

N/A

Screenshot(s)/video(s) showing the problem:

N/A

Request is to include the numpy compile ahead of time so it's already available on the system for any plugin to utilize without having to compile. I believe octopi 0.18 already has the necessary pre-requisites installed for numpy compiling, but to be sure the programs outside of the virtualenv that are required can be installed with the command below for the python 3 requirements.

sudo apt install libgfortran5 libatlas3-base libatlas-base-dev

and then the command to install numpy is within the virtualenv of octoprint source ~/oprint/bin/activate

pip install numpy

I have read the FAQ.

[guysoft]

All those dependencies take a fair amount of space. This is something AFAIK you do once per-install, and only a small subset of people will actually install this. Meaning you are shipping a huge amount of bits no one will use.

What we can do is add an install script.

I am closing this and will help with directions for a variant, unless you can give me a better reason why.

[jneilliii]

With Bed Level Visualizer at nearly 19,000 installs based on the stats I think it might be beneficial to the community over all. That's almost 20% of the known OctoPrint user base.

[guysoft]

@foosel any thoughts?

[foosel]

In principle fine with me, but I'm not so sure it will be as big of a point of frustration with 0.18 as it was so far. The thing is, right now numpy has to be compiled on all instances that need it for one plugin or another. Same btw with PIL. 0.18 ships with a Py3 venv however. Which means we can (and automatically will) finally make use of piwheels which offers all these packages precompiled:

pi@octopiB:~ $ ~/oprint/bin/python --version
Python 3.7.3
pi@octopiB:~ $ ~/oprint/bin/pip install numpy
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Collecting numpy
  Downloading https://www.piwheels.org/simple/numpy/numpy-1.19.2-cp37-cp37m-linux_armv7l.whl (10.5 MB)
     |████████████████████████████████| 10.5 MB 70 kB/s
Installing collected packages: numpy
Successfully installed numpy-1.19.2

Which makes this take around 20s...

[jneilliii]

Fair point @foosel but I have seen where those apt dependencies still have to be installed for python 3 installed version to work.

[guysoft]

@jneilliii When you say "I have seen where those apt dependencies still have to be installed for python 3" - Does that include OctoPi 0.18 nighlty from here?: https://gnethomelinux.com/OctoPi/nightly/
If its not needed then numpy should just get installed each time someone installs the plugin quickly and without compile time, which means that its not needed to pre-ship it.

[foosel]

That's indeed the question. The install quoted above was done against an 0.18 nightly from two weeks or so back that I still had flashed on one of my test Pis. No additional setup that I can think of. However, I haven't confirmed if it works as expected or not, that should be done by someone who has used numpy enough to judge that.

[jneilliii]

Ah, I just checked piwheels and I see that the latest 1.19.2 version is now successful. I think when I was testing this before that version wasn't compiling successfully on piwheels and therefore was downloading and compiling fresh. I will reflash octopi nightly and verify now.

[guysoft]

Cool, update if this solves the problem

[jneilliii]

Ok, so just was able to flash the nightly 0.18 from last night and install the Bed Level Visualizer plugin and although numpy does install quickly now thanks to piwheels.org and python 3 the required apt dependencies still cause errors. This is the specific error I get after install and the plugin does not load.

2020-09-23 18:25:56,201 - octoprint.plugin.core - ERROR - Error loading plugin bedlevelvisualizer
Traceback (most recent call last):
  File "/home/pi/oprint/lib/python3.7/site-packages/numpy/core/__init__.py", line 22, in <module>
    from . import multiarray
  File "/home/pi/oprint/lib/python3.7/site-packages/numpy/core/multiarray.py", line 12, in <module>
    from . import overrides
  File "/home/pi/oprint/lib/python3.7/site-packages/numpy/core/overrides.py", line 7, in <module>
    from numpy.core._multiarray_umath import (
ImportError: libf77blas.so.3: cannot open shared object file: No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/oprint/lib/python3.7/site-packages/octoprint/plugin/core.py", line 972, in _import_plugin
    module = _load_module(module_name, spec)
  File "/home/pi/oprint/lib/python3.7/site-packages/octoprint/plugin/core.py", line 71, in _load_module
    return imp.load_module(name, f, filename, details)
  File "/home/pi/oprint/lib/python3.7/site-packages/octoprint/vendor/imp.py", line 241, in load_module
    return load_package(name, filename)
  File "/home/pi/oprint/lib/python3.7/site-packages/octoprint/vendor/imp.py", line 215, in load_package
    return _load(spec)
  File "<frozen importlib._bootstrap>", line 696, in _load
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/pi/oprint/lib/python3.7/site-packages/octoprint_bedlevelvisualizer/__init__.py", line 7, in <module>
    import numpy as np
  File "/home/pi/oprint/lib/python3.7/site-packages/numpy/__init__.py", line 140, in <module>
    from . import core
  File "/home/pi/oprint/lib/python3.7/site-packages/numpy/core/__init__.py", line 48, in <module>
    raise ImportError(msg)
ImportError: 

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

    https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

  * The Python version is: Python3.7 from "/home/pi/oprint/bin/python3"
  * The NumPy version is: "1.19.2"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: libf77blas.so.3: cannot open shared object file: No such file or directory

After looking into that link, specifically here it appears their are two options for this; either sudo apt install libatlas-base-dev or install the raspberry provided numpy version apt install python3-numpy. I did the first option for libatlas-base-dev (which included the libatlas-base-dev libatlas3-base and libgfortran5) and restarted OctoPrint and then everything was happy. Total increase in storage consumption for that install is listed as 33MB, which to me doesn't seem that heavy to be honest. Let me know your thoughts.

[foosel]

Using the numpy from Raspbian won't work as we don't use the system wide packages but rather the venv, and system wide had caused tons of issues with compatibility in the past.

So I vote for including the libatlas dependency and solve this that way.

[guysoft]

The error says:

ImportError: libf77blas.so.3: cannot open shared object file: No such file or directory

So you don't need the -dev ending packages.
Likely its libatlas-base or something similar like that, because you don't need to build now :)
Any chance you can test that or should I just add it @jneilliii ?

[foosel]

Package libatlas-base is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

E: Package 'libatlas-base' has no installation candidate

Numpy docs say to install the -dev package:

Which depends not on libatlas-base but rather libatlas3-base. Installing that makes numpy at least no longer throw an error on import:

(oprint) pi@octopiB:~ $ python
Python 3.7.3 (default, Jul 25 2020, 13:03:44)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>>

Unsurprisingly so because it actually does contain the missing libf77blas.so.3 file:

(oprint) pi@octopiB:~ $ dpkg -L libatlas3-base
/.
/usr
/usr/lib
/usr/lib/arm-linux-gnueabihf
/usr/lib/arm-linux-gnueabihf/atlas
/usr/lib/arm-linux-gnueabihf/atlas/libblas.so.3.10.3
/usr/lib/arm-linux-gnueabihf/atlas/liblapack.so.3.10.3
/usr/lib/arm-linux-gnueabihf/libatlas.so.3.10.3
/usr/lib/arm-linux-gnueabihf/libcblas.so.3.10.3
/usr/lib/arm-linux-gnueabihf/libf77blas.so.3.10.3
/usr/lib/arm-linux-gnueabihf/liblapack_atlas.so.3.10.3
/usr/share
/usr/share/doc
/usr/share/doc/libatlas3-base
/usr/share/doc/libatlas3-base/README.Debian
/usr/share/doc/libatlas3-base/changelog.Debian.gz
/usr/share/doc/libatlas3-base/changelog.gz
/usr/share/doc/libatlas3-base/copyright
/usr/lib/arm-linux-gnueabihf/atlas/libblas.so.3
/usr/lib/arm-linux-gnueabihf/atlas/liblapack.so.3
/usr/lib/arm-linux-gnueabihf/libatlas.so.3
/usr/lib/arm-linux-gnueabihf/libcblas.so.3
/usr/lib/arm-linux-gnueabihf/libf77blas.so.3
/usr/lib/arm-linux-gnueabihf/liblapack_atlas.so.3

But the -dev package also contains some stuff:

(oprint) pi@octopiB:~ $ dpkg -L libatlas-base-dev | grep so
/usr/lib/arm-linux-gnueabihf/atlas/libblas.so
/usr/lib/arm-linux-gnueabihf/atlas/liblapack.so
/usr/lib/arm-linux-gnueabihf/libatlas.so
/usr/lib/arm-linux-gnueabihf/libcblas.so
/usr/lib/arm-linux-gnueabihf/libf77blas.so
/usr/lib/arm-linux-gnueabihf/liblapack_atlas.so

Checking the dependencies of the python3-numpy package:

(oprint) pi@octopiB:~ $ apt show python3-numpy
Package: python3-numpy
Version: 1:1.16.2-1
Priority: optional
Section: python
Source: python-numpy
Maintainer: Sandro Tosi <morph@debian.org>
Installed-Size: 10.6 MB
Provides: python3-f2py, python3-numpy-abi9, python3-numpy-api13, python3-numpy-dev, python3.7-numpy
Depends: python3 (<< 3.8), python3 (>= 3.7~), python3.7:any, python3:any, libblas3 | libblas.so.3, libc6 (>= 2.27), liblapack3 | liblapack.so.3, python3-pkg-resources
Suggests: gcc (>= 4:4.6.1-5), gfortran, python-numpy-doc, python3-dev, python3-pytest, python3-numpy-dbg
Breaks: python3-aplpy (<< 2.0~rc2-1), python3-astropy (<< 3.1-1), python3-ccdproc (<< 1.3.0-5), python3-dask (<< 1.0.0+dfsg-2), python3-skimage (<< 0.14.1-3), python3-tables (<< 3.4.4-2), python3-theano (<< 1.0.3+dfsg-1)
Homepage: http://www.numpy.org/
Download-Size: 1,873 kB
APT-Sources: http://raspbian.raspberrypi.org/raspbian buster/main armhf Packages
Description: Fast array facility to the Python 3 language
 Numpy contains a powerful N-dimensional array object, sophisticated
 (broadcasting) functions, tools for integrating C/C++ and Fortran
 code, and useful linear algebra, Fourier transform, and random number
 capabilities.
 .
 Numpy replaces the python-numeric and python-numarray modules which are
 now deprecated and shouldn't be used except to support older
 software.
 .
 This package contains Numpy for Python 3.

So that one apparently depends on libblas3 or libblas.so.3. The latter is provided by libatlas3-base:

(oprint) pi@octopiB:~ $ apt show libatlas3-base
Package: libatlas3-base
Version: 3.10.3-8+rpi1
Priority: optional
Section: libs
Source: atlas
Maintainer: Debian Science Team <debian-science-maintainers@lists.alioth.debian.org>
Installed-Size: 11.4 MB
Provides: libblas.so.3, liblapack.so.3
Depends: libc6 (>= 2.27), libgcc1 (>= 1:4.0), libgfortran5 (>= 8)
Conflicts: libcblas3
Breaks: libatlas-base-dev (<< 3.10.3-4~), libblas3 (<< 3.7.1-2~), liblapack3 (<< 3.7.1-2~), libopenblas-base (<< 0.2.20+ds-3~)
Replaces: libcblas3
Homepage: http://math-atlas.sourceforge.net/
Download-Size: 2,399 kB
APT-Manual-Installed: yes
APT-Sources: http://raspbian.raspberrypi.org/raspbian buster/main armhf Packages
Description: Automatically Tuned Linear Algebra Software, generic shared
 ATLAS is an approach for the automatic generation and optimization of
 numerical software. Currently ATLAS supplies optimized versions for the
 complete set of linear algebra kernels known as the Basic Linear Algebra
 Subroutines (BLAS), and a subset of the linear algebra routines in the
 LAPACK library.
 .
 The libraries in this package are built without any processor extension
 instructions, and should run on all processors of this general
 architecture, albeit less than optimally.
 .
 If you want to create an ATLAS packaged optimized for your processor, see the
 section: "Building Optimized Atlas Packages on your ARCH" in README.Debian

but not by libatlas-base-dev:

(oprint) pi@octopiB:~ $ apt show libatlas-base-dev
Package: libatlas-base-dev
Version: 3.10.3-8+rpi1
Priority: optional
Section: libdevel
Source: atlas
Maintainer: Debian Science Team <debian-science-maintainers@lists.alioth.debian.org>
Installed-Size: 20.7 MB
Provides: libblas.so, liblapack.so
Depends: libatlas3-base (= 3.10.3-8+rpi1)
Suggests: libatlas-doc, liblapack-doc
Conflicts: libcblas-dev
Breaks: libatlas-dev (<< 3.10.3-2), libblas-dev (<< 3.7.1-2~), liblapack-dev (<< 3.7.1-2~), libopenblas-dev (<< 0.2.20+ds-3~)
Replaces: libatlas-dev (<< 3.10.3-2), libcblas-dev
Homepage: http://math-atlas.sourceforge.net/
Download-Size: 2,966 kB
APT-Manual-Installed: yes
APT-Sources: http://raspbian.raspberrypi.org/raspbian buster/main armhf Packages
Description: Automatically Tuned Linear Algebra Software, generic static
 ATLAS is an approach for the automatic generation and optimization of
 numerical software. Currently ATLAS supplies optimized versions for the
 complete set of linear algebra kernels known as the Basic Linear Algebra
 Subroutines (BLAS), and a subset of the linear algebra routines in the
 LAPACK library.
 .
 This package includes the headers, the static libraries and symbolic links
 needed for program development.

So based on that I'd say that yes, numpy only needs libatlas3-base, not the -dev package. But maybe we still should go with the suggestions by the official numpy docs.

[guysoft]

@foosel Where do you see that in the docs? I can't see it.

[foosel]

[guysoft]

Ok this page:
https://numpy.org/devdocs/user/troubleshooting-importerror.html#raspberry-pi
The line was added by @seberg here: numpy/numpy@d530d23
I can't find any other history where that came from.

I would say that libatlas3-base makes the most sense because -dev in Debian means its a package source header which we are downloading already pre-built.

[jneilliii]

I appreciate it Guy. That will definitely save me some headaches with support issues in the future. I'll grab the nightly tomorrow after this builds and verify everything is working as expected.

[seberg]

That line came from users with raspbian saying that it solved their issue, it seems indeed like a good idea to drop the -dev from that site if it is not necessary.

[jneilliii]

@guysoft I'm not so sure this is resolved. I just installed 2020-10-07_2020-08-20-octopi-buster-armhf-lite-0.18.0 from the nightly builds and I'm still having the same issue. Is this change already part of the nightly builds because the filename after extraction 2020-08-20-octopi-buster-armhf-lite-0.18.0 makes me believe it might be an older build date being zipped up. After flashing that image and booting up I was getting the same error. SSH to the pi and run sudo apt install libatlas3-base and it indicated 2 new packages would be installed, libatlas3-base and libgfortran5 (dependency). After install everything with numpy worked as expected.

[guysoft]

I use the nighties at http://unofficialpi.org/Distros/OctoPi/nightly/ or http://unofficialpi.org/Distros/OctoPi/nightly-arm64/.

Last build adds this in the build log:

02:04:43 + apt-get -y --force-yes install python3 python3-virtualenv python3-dev git screen subversion cmake avahi-daemon libavahi-compat-libdnssd1 libffi-dev libssl-dev libatlas3-base

Also says:

02:05:58 update-alternatives: using /usr/lib/arm-linux-gnueabihf/atlas/libblas.so.3 to provide /usr/lib/arm-linux-gnueabihf/libblas.so.3 (libblas.so.3-arm-linux-gnueabihf) in auto mode

The reason is that the build you provided has no build date, so I don't know if its old or new

[jneilliii]

Thanks Guy. Not sure why the previous download I got from the other site that went down recently didn't work as expected. The one named "2020-10-22_2020-08-20-octopi-buster-armhf-lite-0.18.0" from this site seems to work perfectly. I really appreciate you adding that.