Wheel support for macOS arm64 (Apple Silicon)

[richiksc]

OpenCV 4.50 supports building for Apple Silicon (view release notes, PR opencv/opencv#18094 was merged and released in 4.50). Now that the size limit for the project on PyPI has been increased, is there any chance we get prebuilt wheels for macOS arm64? The build script supports cross compilation from x86_64 to arm64, so I don't think dedicated Apple Silicon CI hardware is needed.

[skvark]

Everything is pending this largeish CI migration task described in here: #422

This repository does not use those build scripts but it looks like the cross compilation needs just a few additional CMake flags so it shouldn't be a problem to build those wheels in the future.

[richiksc]

@skvark Thanks! Looks like Travis CI shutting off OSS repositories has significantly interfered with any work.

[skvark]

One important thing that I would like to add to my previous comment:

It will take some time to get native numpy packages for Apple silicon. In practice, this means months, maybe even longer. This package depends on numpy which means that native arm64 for Apple silicon won't be possible before a compatible numpy version has been released. Requires also Apple Silicon CI runners unless cross compilation is used.

Recommended reading: https://github.com/neurolabusc/AppleSiliconForNeuroimaging

Numpy: numpy/numpy#17807 (comment)

[richiksc]

@skvark In regards to cross compilation, I don't know whether numpy are using macOS CI runners are not, but Clang 12+ on macOS with the Xcode 12+ Command Line Tools installed supports cross compilation from Intel to arm64-apple-darwin20.2.0.

[ogrisel]

It will take some time to get native numpy packages for Apple silicon.

Note that the conda-forge community project had built a M1-native version of the numpy, scipy, pandas, scikit-learn... stack even prior to the release of the first M1 Mac Books. You can install those packages by setting up a conda env with:

https://github.com/conda-forge/miniforge

I use it on a daily basis and it works great. Note that conda-forge does not ship tensorflow though.

[skvark]

It will take some time to get native numpy packages for Apple silicon.

Note that the conda-forge community project had built a M1-native version of the numpy, scipy, pandas, scikit-learn... stack even prior to the release of the first M1 Mac Books. You can install those packages by setting up a conda env with:

https://github.com/conda-forge/miniforge

I use it on a daily basis and it works great. Note that conda-forge does not ship tensorflow though.

This is not a conda-forge project. This repository uses the standard Python package distribution tooling (pip, wheel and manylinux just to name a few). I cannot mix conda packages to these packages because conda uses their own custom package management system. Conda-forge has also a very large community and more resources at their disposal.

In short: when someone runs pip install opencv-python, pip must be able to download and install compatible numpy version from PyPI. Same applies to the build toolchain in this repository.

[richiksc]

@svkark numpy has a new release, v1.19.5, that seems to build on M1. numpy/numpy#17807 (comment)

[skvark]

This repository depends on the official binary releases which means that I need official pre-built wheel packages. There are none: https://pypi.org/project/numpy/#files

You can of course run the build for both numpy and opencv-python now manually and start using native versions.

[asmorkalov]

OpenCV team considers to migrate some of package builds to Github Action platform, first of all for Mac. M1 packages availability depends on M1 support on Github side. Related items:

• Github actions ticket about M1 support: Support for VMs on Apple M1 actions/runner-images#2187
• https://github.com/github/roadmap/projects/1

[asmorkalov]

More details: github/roadmap#95

[hannesa2]

My motivation is a working pip3 install opencv-python on Apple Silicon M1

When I understand right, it makes no sense to donate some Travis credits to make a single event macOS build for arm64 (M1), because a CI is needed with Apple Silicon M1 machines. Travis has no such machines. Am I right ?

Github actions doesn't have any such machines either. That's why a CI with Github action doesn't fulfill my needs too.
Anyway a Github CI would make sense to be ready prepared, when they would be available.
(I started to make the first steps on my fork, but I need some help)

You can of course run the build for both numpy and opencv-python now manually and start using native versions.

To build local opencv-python is currenlty the only way to use OpenCV with Python on Apple Silicon. Am I right ?

Using https://github.com/apple/tensorflow_macos install numpy, so I guess, this is no blocker anymore

[hannesa2]

I tried to build local

git submodule add https://github.com/skvark/opencv-python.git
git submodule update --init --recursive
cd opencv-python
ENABLE_CONTRIB=0 
ENABLE_HEADLESS=1
pip wheel . --verbose

but I run into
ERROR: Command errored out with exit status 1: /Users/hannes/git/picar-ML/udacity-simulator/tensorflow_venv2/bin/python3 /Users/hannes/git/picar-ML/udacity-simulator/tensorflow_venv2/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /private/var/folders/np/7ckckk8s5fd169kvnnfhjkk40000gr/T/pip-build-env-3zg4n104/overlay --no-warn-script-location -v --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools wheel scikit-build cmake pip 'numpy==1.13.3; python_version=='"'"'3.6'"'"' and platform_machine != '"'"'aarch64'"'"'' 'numpy==1.19.3; python_version>='"'"'3.6'"'"' and sys_platform == '"'"'linux'"'"' and platform_machine == '"'"'aarch64'"'"'' 'numpy==1.14.5; python_version=='"'"'3.7'"'"' and platform_machine != '"'"'aarch64'"'"'' 'numpy==1.17.3; python_version=='"'"'3.8'"'"' and platform_machine != '"'"'aarch64'"'"'' 'numpy==1.19.3; python_version>='"'"'3.9'"'"' and platform_machine != '"'"'aarch64'"'"'' Check the logs for full command output.

Does is mean numpy is still an issue. Any chance to solve this ?

[skvark]

Using Travis is way too expensive and I'm pretty sure that it will not be used in any future builds. You are right that there is no Apple Silicon build environment in Travis. And yes, you have to do a local build of OpenCV for Apple Silicon. Another option is to use Rosetta but I don't know how that works (https://developer.apple.com/documentation/apple-silicon/about-the-rosetta-translation-environment).

NumPy doesn't provide a native wheel for Apple Silicon: https://pypi.org/project/numpy/1.20.1/#files

So yes, we need native Apple Silicon wheels to PyPI for NumPy before this project can be built and published to PyPI. If the NumPy wheels are not available, pip install opencv-python will fail on Apple Silicon unless pip is able to compile NumPy and CMake during install for Apple Silicon, and to my understanding that is not currently possible.

You can manually build the native NumPy wheel or find some pre-built archive. To build NumPy, please see the NumPy issue: numpy/numpy#17807 (comment)

Then you need to tell pip that you will not be using NumPy from PyPI but from your local custom installation (see for example https://stackoverflow.com/a/15031843/5795428). The next issue will be cmake, which does not provide native Apple Silicon wheels either (https://pypi.org/project/cmake/#files). You will need to build it also locally for Apple Silicon.

I don't own any Apple machines so I can't help much with this.

[hannesa2]

I followed numpy/numpy#17807 (comment)

git submodule add git@github.com:numpy/numpy.git
cd numpy
OPENBLAS="$(brew --prefix openblas)" MACOSX_DEPLOYMENT_TARGET=11.1 python3 -m pip install cython --no-use-pep517
OPENBLAS="$(brew --prefix openblas)" MACOSX_DEPLOYMENT_TARGET=11.1 python3 -m pip install numpy --no-use-pep517
OPENBLAS="$(brew --prefix openblas)" MACOSX_DEPLOYMENT_TARGET=11.1 python3 -m pip install pandas --no-use-pep517
OPENBLAS="$(brew --prefix openblas)" MACOSX_DEPLOYMENT_TARGET=11.1 python3 -m pip install pybind11 --no-use-pep517
OPENBLAS="$(brew --prefix openblas)" MACOSX_DEPLOYMENT_TARGET=11.1 python3 -m pip install scipy --no-use-pep517
brew install libjpeg zlib
python3 -m pip install pillow
python3 -m pip install matplotlib
OPENBLAS="$(brew --prefix openblas)" MACOSX_DEPLOYMENT_TARGET=11.1 python3 -m pip install scikit-learn --no-use-pep517\n
OPENBLAS="$(brew --prefix openblas)" MACOSX_DEPLOYMENT_TARGET=11.1 python3 -m pip install statsmodels --no-use-pep517\n
cd ../opencv-python
pip wheel . --verbose
pip install numpy --no-index --find-links file:///Users/hannes/git/picar-ML/udacity-simulator/numpy/numpy

all without errors, bun on last I see this

Looking in links: file:///Users/hannes/git/picar-ML/udacity-simulator/numpy/numpy
Requirement already satisfied: numpy in /Users/hannes/git/picar-ML/udacity-simulator/tensorflow_venv2/lib/python3.8/site-packages (1.18.5)

and pip wheel . --verbose still runs into same error 😞

[togume]

👋 got here after going down the rabbit hole to install Auto Editor.

numpy 1.20.1 now builds natively for M1/aarch64, so it should be possible to remove that as a blocker for this.

I'm now going down the rabbit hole to install 4.5.1.48 manually. Wish me luck.

Update: mental horsepower down and stopped because I'm a noob when it comes to manually installing with python. I'll keep trying. In case anyone has instructions on how to manually install https://github.com/opencv/opencv-python/releases/tag/48 on macOS M1, that'd be swell. Otherwise, I'll continue documenting my process.

[hannesa2]

@togume I don't want eliminate your motivation. Having a pip installation would be very nice !

I at least gave up here with my too small knowledge, but I made it work Tensorflow with openCV on Apple Silicon M1

[skvark]

Easiest way to try if numpy builds correctly or has been built & installed beforehand:

Follow the manual build instructions in README until step 5 (https://github.com/opencv/opencv-python#manual-builds)

Replace step 5. with a manual install of build dependencies: pip install scikit-build cmake (add numpy here if it is not installed yet)

After that run: python setup.py bdist_wheel

I recommend doing these steps in a fresh virtualenv with up-to-date tools.

If that does work, then we need to update pyproject.toml to reflect the Apple Silicon configuration. After that just a clone of this repository and pip install opencv-python should do it.

[togume]

Thanks, @skvark! TL;DR: your instructions worked and I was able to successfully build.

Doing. Here's my log:

• Using asdf to manage Python versions, and using freshie of 3.9.2
• numpy successfully installed directly with pip install numpy beforehand
• pip install scikit-build cmake installed successfully
  • Got error because I didn't do --recursive on the git clone, which didn't grab deps like cv2 --> re-clone correctly
• python setup.py bdist_wheel worked and I now have opencv_python-4.5.1.48-cp39-cp39-macosx_11_2_arm64.whl in dist!

Ran out of time. Now I have to figure out how to use it with pip install auto-editor so it's picked up :). More to come!

[togume]

Amazing what not being tired allows you to do... 🤦

TIL that after you have the module built locally, you do python setup.py install and voila, you have the module available system wide.

Confirming that I was able to get auto-editor working with all dependencies, including opencv.

Thanks for all the support, people!

Let me know if there's any feedback, questions, further testing I can help with here.

[togume]

Nice, I follow your steps with Github actions
I will apply the your new steps local and there too.

I want to ask you, how do you verify auto-editor working with all dependencies, including opencv ?

@hannesa2 because it installed completely with pip install auto-editor and then it was available in my path. Then I used it against a video, and it worked as expected!

Regarding the tensorflow error, I'm not sure because I haven't played with it.

[skvark]

@hannesa2 You are trying to install Tensorflow Python 3.8 wheels to Python 3.9 virtualenv. Installing Tensorflow is not related to opencv-python so please use e.g. Stack Overflow or Tensorflow issue tracker for those questions.

[therain7]

Hi. Any ideas how to fix these?

P. S. Compiled opencv from sources

[therain7]

Tried several times with different versions of numpy and opencv. Didn't work, same error every time. Switched to miniforge conda eventually. Their opencv works perfectly

[luisdemarchi]

sorry for the ignorance, but what do i need to do after i generate the zipped file?

pip install ./dist/opencv_python-4.5.2+2344814-cp39-cp39-macosx_11_4_arm64.whl
ERROR: opencv_python-4.5.2+2344814-cp39-cp39-macosx_11_4_arm64.whl is not a supported wheel on this platform.

[asenyaev]

sorry for the ignorance, but what do i need to do after i generate the zipped file?

pip install ./dist/opencv_python-4.5.2+2344814-cp39-cp39-macosx_11_4_arm64.whl
ERROR: opencv_python-4.5.2+2344814-cp39-cp39-macosx_11_4_arm64.whl is not a supported wheel on this platform.

You have to change the name of a wheel from opencv_python-4.5.2+2344814-cp39-cp39-macosx_11_4_arm64.whl to opencv_python-4.5.2+2344814-cp39-cp39-macosx_11_0_arm64.whl, because all python versions (3.7, 3.8, 3.9) has the latest compatible tag only for 11.0 version of MacOS (macosx_11_0_arm64 ), that's why the wheel with 11.1 version cannot be installed
(how to check compatible tags: python3.7 -m pip debug -v | grep -A 10 'Compatible tags').

Another solution is to define the variable MACOSX_DEPLOYMENT_TARGET before building a wheel: export MACOSX_DEPLOYMENT_TARGET=11.0

[asmorkalov]

PR: #507. Package will be available with OpenCV 4.5.3 release.