Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ARM support #13

Closed
skvark opened this issue Sep 6, 2016 · 37 comments
Closed

ARM support #13

skvark opened this issue Sep 6, 2016 · 37 comments

Comments

@skvark
Copy link
Member

skvark commented Sep 6, 2016

ARM builds would be needed to support for example Raspberry Pi.

@skvark
Copy link
Member Author

skvark commented Sep 6, 2016

...but manylinux tag does not support ARM?

@gkvoelkl
Copy link

Thanks for making that great Python installer.

When will there be a version for raspberry pi?

Best regards

Gerhard

@skvark
Copy link
Member Author

skvark commented Jan 29, 2017

Pre-built wheel distributions which target ARM based Linux devices can't be done with current toolchains. The PEP (https://www.python.org/dev/peps/pep-0513/) which defines the manylinux tag recognizes only certain architectures (x86, i686).

Please see also this issue: pypa/manylinux#84

However, Python's packaging system is evolving all the time and this might be possible at some point in the future.

@gkvoelkl
Copy link

I hope that time will come fast.

I have seen some packages that could be installed with pip

Examp. https://github.com/samjabrahams/tensorflow-on-raspberry-pi

Do they create their own toolchain?

@skvark
Copy link
Member Author

skvark commented Jan 30, 2017

Yes, they have created their own special Docker image with all the correct dependencies and it runs on actual RPi hardware. Tensorflow itself has some tools to generate the package after the build.

What I would need to make this happen:

  • similar Docker image as the Cent OS 5 manylinux image is but with some RPi /ARM baseline which is common for all devices - this will probably require QEMU based image or some image with cross-compilation toolkit like Scratchbox2
  • I must be able to run the images on Travis
  • afaik the resulting wheels cannot be distributed via PyPI so I would need to host them manually somewhere

This is not a task that can be done over a weekend, it will require most probably several weeks, maybe months, of work.

@illume
Copy link
Contributor

illume commented Feb 23, 2018

Raspberry pi have a build system where they try to build all the pypi packages -- piwheels.

piwheels website: https://www.piwheels.hostedpi.com/
blogpost about what it is: http://bennuttall.com/piwheels-building-a-faster-python-package-repository-for-raspberry-pi-users/
FAQ: https://www.piwheels.hostedpi.com/faq.html

They are receptive to PRs to include Debian dependencies here: https://github.com/bennuttall/piwheels/

I guess all your dependencies aren't in the Debian they use though? Also, compiling on real hardware can take a long time for big C++ libraries.

@sctv
Copy link

sctv commented Apr 5, 2018

In the latest raspian it automatically downloads the piwheels if they exist.
(opencv-python piwheels for rpi arm)] for python 3, no python 2. Is that a problem or a feature?

@skvark
Copy link
Member Author

skvark commented Apr 5, 2018

I have no idea. I do not maintain those packages and I don't know how they were created.

@bennuttall
Copy link

We haven't built Python 2 wheels - we might do in future. But there's not much future left.

@sctv
Copy link

sctv commented Apr 12, 2018

@bennuttall only use case I have for python2 wheels is for dependencies for Inkscape extensions, as Inkscape still uses python2.

@popjxc
Copy link

popjxc commented Apr 19, 2018

I try to build this project on QEMU based docker image, it quite slow and often make docker build timeout. it's really painful...

@cooldeaaap1
Copy link

when Can I "pip install opencv_contrib_python" in Raspberry Pi?

@skvark
Copy link
Member Author

skvark commented Jul 19, 2018

PyPi does not support ARM wheels yet. I have no idea if it will get support for ARM wheels and if it does when it happens.

@bennuttall
Copy link

It does! But I wouldn't recommend publishing Arm wheels, it's a bit messy. See piwheels/piwheels#66

Raspberry Pi users can use piwheels (configured by default in Raspbian) to get pre-built packages which work on all Raspberry Pi models. We just haven't built for Python 2 so far.

@waveform80
Copy link

I should add, I recently added the *-headless builds from this repo too (thanks to @skvark's fine work!), so you can "pip install opencv_contrib_python_headless" too. I watch this repo, so when there's a new release I try to add it to piwheels as soon as I'm able, but there'll usually be some delay.

@cooldeaaap1
Copy link

@bennuttall I am sorry that :
I have use pip install opencv_contrib_python-3.2.0.7-cp35-cp35m-linux_armv7l.whl and It seems OK, but when I
python import cv2
It told me that:
Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/pi/.pyenv/versions/3.5.2/lib/python3.5/site-packages/cv2/__init__.py", line 7, in <module> from . import cv2 ImportError: libhdf5_serial.so.100: cannot open shared object file: No such file or directory
even
sudo apt-get update sudo apt-get install libhdf5-dev sudo apt-get update sudo apt-get install libhdf5-serial-dev
It didnot work ...
How can I makr it work ?

@waveform80
Copy link

@cooldeaaap1 libhdf5-dev should be pulling in libhdf5-100 which in turn provides /usr/lib/arm-linux-gnueabihf/libhdf5_serial.so.100 (you don't actually need the *-dev packages for installation - those are for building - but they pull in the underlying library anyway, so no harm).

@waveform80
Copy link

@cooldeaaap1 when you say sudo apt-get update; sudo apt-get install libhdf5-dev libhdf5-serial-dev didn't work, do you mean installing them didn't work, or importing cv2 after installing them didn't work? If the former, then I'd need to see the error from apt-get. If the latter I'd have to ask if you were running the latest Raspbian or not?

@cooldeaaap1
Copy link

@waveform80
Thank for your help , I change to the latest Raspbian , I want opencv-contrib-python==3.2.0.7;but I got error

ImportError: /usr/local/lib/python3.5/dist-packages/cv2/cv2.cpython-35m-arm-linux-gnueabihf.so: undefined symbol: hb_buffer_get_glyph_infos
change to others ,like 3.3.0.9 ,it works, Is it a bug ?

@walchko
Copy link

walchko commented Jul 29, 2018

PyPi does not support ARM wheels yet.

@skvark, thanks for the great work. I am confused when you say pypi doesn't support arm, it does. Unless I am misunderstanding you, if you look at something I published sslam-0.0.3-cp27-cp27mu-linux_armv7l.whl it is a wheel file for armv7. I did nothing special, here is my setup.py, just a simple c extension. I did build the package on a real raspberry pi, not travis, because that is only x86. Also, my stuff is trivial compared to opencv. Again, I might be misunderstanding your comment. 😄

@waveform80
Copy link

@cooldeaaap1 could be a bug in 3.2 (there's something vaguely similar mentioned in opencv/opencv_contrib#1175) but if 3.3 works then I'd just stick with that (or later).

@walchko See last FAQ at https://www.piwheels.org/faq.html - it's slightly out of date in that pypi.org is now the "next gen" PyPI that's talked about but the rest of it still stands (there's tons of ARM architectures and stuff compiled on Raspbian on a Pi isn't "true" ARMv7 because it still has to be compatible with the ARMv6 userland).

@cooldeaaap1
Copy link

@waveform80 I am sorry that : I had better to use "3.2.0.7" ,because I want to use a prject https://github.com/AirtestProject/Airtest
and it's requirements contains "opencv-contrib-python 3.2.0.7" https://github.com/AirtestProject/Airtest/blob/master/requirements.txt

image

@skvark
Copy link
Member Author

skvark commented Jul 30, 2018

@skvark, thanks for the great work. I am confused when you say pypi doesn't support arm, it does. Unless I am misunderstanding you, if you look at something I published sslam-0.0.3-cp27-cp27mu-linux_armv7l.whl it is a wheel file for armv7. I did nothing special, here is my setup.py, just a simple c extension. I did build the package on a real raspberry pi, not travis, because that is only x86. Also, my stuff is trivial compared to opencv. Again, I might be misunderstanding your comment. 😄

Yeah, you can upload arbitrary ARM builds to PyPI nowadays. My comment was a bit misleading. What I mean is that there is no universal support for different ARM based boards. Yes, I could start building these packages for ARM on a Raspberry Pi but that does not mean that the packages are compatible with other boards.

Therefore, it does not make any sense for this repository to start supporting one product line (RasPi) as there are many different ARM based boards. For example almost all mobile phones which are running Android are ARM based (!). It would be cool to run pip install opencv-python on my Sailfish OS phone, but that's not just yet possible with the current state of Python package management. Please see pypa/manylinux#84.

However, thanks to @bennuttall's and @waveform80's great work, you can use pre-built wheels on Raspberry Pi's. For more information, see https://www.piwheels.org/.

@Veilkrand
Copy link

Veilkrand commented Sep 11, 2018

@skvark I'm in the same situation described before. After installing opencv-contrib-python-3.4.2.17 using piwheels and importing cv2 in python3 it shows:
ImportError: libhdf5_serial.so.100: cannot open shared object file: No such file or directory

I have tried to reinstall libhdf5 and libhdf5_serial but it didn't resolved the problem. I'm using Ubuntu Mate in RPI. Any insights? Thanks

@skvark
Copy link
Member Author

skvark commented Sep 11, 2018

I don't maintain Raspberry Pi builds. Please open an issue to https://github.com/bennuttall/piwheels.

@native-api
Copy link
Contributor

Can Travis build for ARM?

According to travis-ci/travis-ci#2790, only cross-compiling to Android is available out of the box.

https://github.com/Pi4J/pi4j/blob/master/.travis.yml is an example showing it's possible to cross-compile for Pi with a downloaded "rpi-tools" package.

As per Running Travis CI tests on ARM architecture | tomaz.me, it's also possible to run tests with qemu.

Finally, https://bennuttall.com/piwheels-building-a-faster-python-package-repository-for-raspberry-pi-users/ and https://www.piwheels.hostedpi.com/ say that the piwheels project compile everything from PyPI (though, if the latter is correct, only for py3.4 and py3.5 -- looks strange, maybe the info is outdated). So all we need to do is provide an sdist that they can run pip wheel <pkg> on.

@waveform80
Copy link

Finally, https://bennuttall.com/piwheels-building-a-faster-python-package-repository-for-raspberry-pi-users/ and https://www.piwheels.hostedpi.com/ say that the piwheels project compile everything from PyPI (though, if the latter is correct, only for py3.4 and py3.5 -- looks strange, maybe the info is outdated). So all we need to do is provide an sdist that they can run pip wheel on.

We attempt to build everything from PyPI (and keep everything that succeeds :). As OpenCV is a little more tricky (as you noted there's no sdist that works "out of the box") we handle that separately; specifically I watch this repo and when @skvark makes a new release I build them on my Pis and import them into piwheels when they're done (usually a couple of days later!). Tensorflow is also handled separately (specifically, the maintainers send Ben the pre-built wheels and he imports them).

We only handle py34 and py35 at the moment because we (currently) only support Raspbian Jessie (py34) and Raspbian Stretch (py35). At some point, when I've got a spare couple of weekends, we'll add py27 to that mix but it's not a trivial addition (the build queue logic is already fairly complex and multiple major versions makes things even more complex).

We probably won't be supporting anything more recent until Raspbian Buster makes an appearance (there was a vague attempt at supporting py36 via Ubuntu images on the builders, but that just caused more trouble than it was worth because some py36 users had built their own on Raspbian, some were using Ubuntu, and thus each had different lib versions installed so we're sticking strictly to Raspbian for now).

@waveform80
Copy link

Anyway, as noted above - if you've any issues with the piwheels builds, do feel free to open issues at https://github.com/bennuttall/piwheels

@bennuttall
Copy link

To resolve the "cannot open shared object file" error, see https://blog.piwheels.org/how-to-work-out-the-missing-dependencies-for-a-python-package/

@native-api
Copy link
Contributor

Travis now supports natively building for the ARM architecture: https://blog.travis-ci.com/2019-10-07-multi-cpu-architecture-support

It's currently in alpha so be aware of possible problems.

@illume
Copy link
Contributor

illume commented Oct 20, 2019

In particular, no language: python support at the moment (so you have to build/install python yourself and such).

@skvark skvark mentioned this issue Dec 21, 2019
@skvark skvark changed the title Can Travis build for ARM? ARM support Jun 22, 2020
@Red-Eyed
Copy link

I wonder, why not to use cross compilation?

@AshokBhat
Copy link

AshokBhat commented Jul 12, 2020

@skvark There is an option to get Arm servers via the WorksOnArm project. Please see the https://github.com/WorksOnArm/cluster/issues/209 for details. Tagging @vielmetti for visibility.

@vielmetti
Copy link

There has also been some progress in getting wheels built for some Python projects for arm64 - notably this report from numpy at https://twitter.com/pypackages/status/1274444742621888513 .

There's a couple of options through the Works on Arm effort mentioned above:

  • shared CI on Drone Cloud w/native arm64 support
  • shared CI on Travis CI w/native arm64 support
  • dedicated CI on a dedicated machine directly through Works on Arm

the preference for dedicated systems is for projects that have some compute density needed - ideally you'd want to build more than one package, or be working on a system that is a critical piece not yet ported but that lots of other things depend on, etc. Much of the time, Drone or Travis is a more effective way of getting CI done than dedicated systems.

@daun-io
Copy link

daun-io commented Dec 7, 2020

Hello, I guess this entire thread is yet focused on embedded devices such as raspberry pi however is there any further plan to support ARM64 architecture and corresponding wheels for new Apple silicon since there is no wheel compiled for macOS / arm64? Thanks!

@skvark
Copy link
Member Author

skvark commented Dec 7, 2020

Please use the search function, there is a dedicated issue for arm64.

@skvark
Copy link
Member Author

skvark commented Jan 5, 2021

The latest release includes experimental arm64 (aarch64) Linux wheels. Older arm variants will not be supported. Please use PiWheels or compile from source if you need armv7 / armv6 wheels.

Linux arm64 (aarch64) tracking issue: #366
Apple Silicon arm64 tracking issue: #429

@skvark skvark closed this as completed Jan 5, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests