Native wheels for the macos/arm64 platform (Apple Silicon M1 hardware) #19137

ogrisel · 2021-01-08T12:56:07Z

Making this happen depends on the resolution of the following upstream issues:

BLD: fail to build on Apple M1 numpy/numpy#17807 (building numpy in native mode on M1)
Segmentation fault on import of scipy.integrate on Apple M1 ARM silicon scipy/scipy#13364 (segfault on scipy.integrated on M1 with openblas and gfortran from homebrew)
public release of scipy with a wheel for the macos/arm64 platform
Support for macOS universal2 builds for ARM-based Macs pypa/cibuildwheel#473 (comment) discussion on how to add support for building M1 native wheels for Python packages on public CI (e.g. github actions, azure pipelines and co).
update the Wheel build github actions workflow in the scikit-learn repo to enable building for the new platform using cross-compilation on intel macos executors.

Note: macos/arm64 is also known as macos/aarch64.

The text was updated successfully, but these errors were encountered:

ogrisel · 2021-01-08T12:57:44Z

In the mean time, users are recommended to use the macos/arm64 packages of conda-forge by using the miniforge or mambaforge installers:

https://github.com/conda-forge/miniforge

ogrisel · 2021-02-19T14:00:18Z

Note that we could provide both macos/arm64 wheels and universal2 wheels, assuming that numpy and scipy do it.

universal2 wheels for numpy and scipy might be challenging to generate because those vendor openblas which has to be partially built by gfortran which cannot natively generate fat universal2 binaries. scipy also has other fortran compiled extensions (l-bfgs, arpack....). An additional merge operation would have to be performed to generate such fat binaries.

For scikit-learn, we do not need to build anything in fortran and clang can natively generate fat binaries for C/C++ compiled extensions (using cibuildwheel).

UnixJunkie · 2021-06-09T05:00:05Z

Forcing people to use conda/miniconda. Oh my god...
conda: the worst package manager I have ever encountered in my life (apart from an install from scratch, it doesn't know how to do anything in a timely manner).

PS: thanks for the workaround though

grisaitis · 2021-06-22T18:55:22Z

conda: the worst package manager I have ever encountered in my life

Haha, have you never used pipenv? Or the R programming language?

Be grateful, very grateful.

UnixJunkie · 2021-06-23T00:36:58Z

I don't know pipenv.

PS: The R package manager is pretty good, in my experience.

jnothman · 2021-06-24T15:00:48Z

Try mamba. It's a more timely conda

anirudh-g · 2021-09-02T04:26:26Z

Hi All. Is there any update on this issue?

csp197 · 2021-10-05T21:32:18Z

@ogrisel
Scipy provides nightly wheel builds for arm64 - scipy/scipy#13409 (comment)

Is this something that sklearn can provide as well?
Conda workaround is a good temp fix, but not great for long term...

Mantower · 2021-10-06T12:46:12Z

Hi.

I found this fix that seems to install scikit without the need of using conda

# SciPy:
python -m pip install --no-cache --no-use-pep517 pythran cython pybind11 gast"==0.4.0"
pyenv rehash
python -m pip install --no-cache --no-binary :all: --no-use-pep517 scipy"==1.7.1"

# Scikit-Learn
python -m pip install --no-use-pep517 scikit-learn"==0.24.2"

In my case the scikit line was sufficient as I had already installed cython and scipy beforehand.
Good luck!

Edit: A thought I just had; it might be that you can change the version number to 1.0.0, but this isn't something I've tested myself.

appleseedexm · 2021-10-08T11:02:42Z

Hi.

I found this fix that seems to install scikit without the need of using conda
[...]

This lead me on the right path, however I had to install nightly scipy instead. Afterwards I was able to install scikit-learn 1.0.0 without a problem:

# SciPy:
python -m pip install --no-cache --no-use-pep517 pythran cython pybind11 gast"==0.4.0"
pyenv rehash
python -m pip install --pre -i https://pypi.anaconda.org/scipy-wheels-nightly/simple scipy

# Scikit-Learn
python -m pip install --no-use-pep517 scikit-learn"==1.0.0"

HRezaei · 2021-10-17T09:36:44Z

Hi.
I found this fix that seems to install scikit without the need of using conda
[...]

This lead me on the right path, however I had to install nightly scipy instead. Afterwards I was able to install scikit-learn 1.0.0 without a problem:
# SciPy:
python -m pip install --no-cache --no-use-pep517 pythran cython pybind11 gast"==0.4.0"
pyenv rehash
python -m pip install --pre -i https://pypi.anaconda.org/scipy-wheels-nightly/simple scipy

# Scikit-Learn
python -m pip install --no-use-pep517 scikit-learn"==1.0.0"

For me it worked even without pyenv rehash, I hadn't pyenv installed.

MikhailKravets · 2021-10-18T15:24:06Z

Finally, I fixed this problem for myself. I was able to build scikit-learn. You may find the wheel and the instruction on how to install scikit-learn on M1 in this repo.

My problem was in scipy. I couldn't build it myself. But the guys from Anaconda did it 🎉 .

So installation instruction. Pipenv

Look for required scipy in Anaconda's pypi
Download scipy from Anaconda's pypi
Download scikit-learn from this repository
Place them at .wheels directory of your project
Edit your Pipfile to have these lines included

scipy = {versions = "*", markers = "platfotm_machine != 'arm64'"}
scipy-m1 = {path = ".wheels/scipy-1.8.0.dev0+1753.a063cf3-cp38-cp38-macosx_11_0_arm64.whl", markers = "platfotm_machine == 'arm64'"}

scikit-learn = {versions = "*", markers = "platfotm_machine != 'arm64'"}
scikit-learn-m1 = {path = ".wheels/scikit_learn-1.1.dev0-cp39-cp39-macosx_11_0_universal2.whl", markers = "platfotm_machine == 'arm64'"}

Run pipenv install --skip-lock && pipenv lock --pre

Installation instruction. Pip

Download scikit-learn from this repository
Place it at .wheels directory of your project
Install scipy from Anaconda

pip install scipy --index-url=https://pypi.anaconda.org/scipy-wheels-nightly/simple

Install scikit-learn

pip install .wheels/scikit_learn-1.1.dev0-cp39-cp39-macosx_11_0_universal2.whl

sunilshah · 2021-10-18T20:47:06Z

I have homebrew installed python 3.9.7 and scipy version 1.7.1.

I had a version of scikit-learn installed from dev 1.0 source of March 3, 2021.

I installed versions 1.0

pip3 install -U --no-use-pep517 scikit-learn

It works fine.

However, in my testing of linear_model, compared to the same version running on Rosetta2, the native code on M1 runs 1.67x slower! Almost all other numerical codes run faster in native mode.

nimafanniasl · 2021-10-26T17:04:38Z

i have an proplem to install sklearn:

PS C:\Users\Nima> pip install -U scikit-learn
Collecting scikit-learn
  Downloading scikit-learn-1.0.1.tar.gz (6.6 MB)
     |████████████████████████████████| 6.6 MB 136 kB/s
  Installing build dependencies ... -

it is staying in here!

python 3.10.0 64-bit

danieltomasz · 2021-10-26T23:27:26Z

There was a problem with scipy with 3.10, maybe trying with 3.9 will help scipy/scipy#13102 (comment)
Also try upgrading pip, sometimes it helps with new wheels

ShashankKrishnaV · 2021-10-27T08:34:42Z

Hi.
I found this fix that seems to install scikit without the need of using conda
[...]

This lead me on the right path, however I had to install nightly scipy instead. Afterwards I was able to install scikit-learn 1.0.0 without a problem:
# SciPy:
python -m pip install --no-cache --no-use-pep517 pythran cython pybind11 gast"==0.4.0"
pyenv rehash
python -m pip install --pre -i https://pypi.anaconda.org/scipy-wheels-nightly/simple scipy

# Scikit-Learn
python -m pip install --no-use-pep517 scikit-learn"==1.0.0"

You are literally a saviour for me. I am trying it for nearly 4 hrs now.

SiddyP · 2021-11-01T07:39:52Z

Can confirm this is working on M1 (python 3.8, nightly build of scipy)
pip3 install -U --no-use-pep517 scikit-learn

If you have issues make sure you also have Cython >= 0.28.5 installed.

xiangpeng2008 · 2021-11-03T08:26:11Z

Can confirm this is working on M1 (python 3.8, nightly build of scipy) pip3 install -U --no-use-pep517 scikit-learn

If you have issues make sure you also have Cython >= 0.28.5 installed.

pip3 install -U --no-use-pep517 scikit-learn doesn't work for me, but python -m pip install --no-use-pep517 scikit-learn"==1.0.0" works

MarcoTerzulli · 2021-11-04T18:17:17Z

Can confirm this is working on M1 (python 3.8, nightly build of scipy) pip3 install -U --no-use-pep517 scikit-learn

If you have issues make sure you also have Cython >= 0.28.5 installed.

Unfortunately, it's not working for me. I'm on Python 3.9 and M1 Pro. I'm still getting the same error.

I hope there's a way for installing it without needing to use Conda.

EDIT: I managed to install it using the method posted in MikhailKravets comment above.

danieltomasz · 2021-11-04T18:20:05Z

Can confirm this is working on M1 (python 3.8, nightly build of scipy) pip3 install -U --no-use-pep517 scikit-learn
If you have issues make sure you also have Cython >= 0.28.5 installed.

Unfortunately, it's not working for me. I'm on Python 3.9 and M1 Pro. I'm still getting the same error.

I hope there's a way for installing it without needing to use Conda.

Do you have the latest pip?

MarcoTerzulli · 2021-11-04T18:24:16Z

Can confirm this is working on M1 (python 3.8, nightly build of scipy) pip3 install -U --no-use-pep517 scikit-learn
If you have issues make sure you also have Cython >= 0.28.5 installed.

Unfortunately, it's not working for me. I'm on Python 3.9 and M1 Pro. I'm still getting the same error.
I hope there's a way for installing it without needing to use Conda.

Do you have the latest pip?

Yes, just checked

MarcoTerzulli · 2021-11-04T18:30:58Z

Installation instruction. Pip

Download scikit-learn from this repository

Place it at .wheels directory of your project

Install scipy from Anaconda
pip install scipy --index-url=https://pypi.anaconda.org/scipy-wheels-nightly/simple
Install scikit-learn
pip install .wheels/scikit_learn-1.1.dev0-cp39-cp39-macosx_11_0_universal2.whl

I finally managed to install it. I have already tried this morning, but probably I messed something.
Python 3.9, latest version of pip, and M1 Pro

Karan-S-Mittal · 2021-11-08T13:55:49Z

thanks this works for me. and using miniforge is also quite similar.
I had some libraries which were depended on scipy so installing it this way, it makes other pip installs very easy on m1.

matteosantama · 2021-11-13T22:00:10Z

Do the maintainers plan to backport M1 functionality to sklearn<1.0.0? In other words, should M1 users expect to only be able to use sklearn version 1.0.0 and later?

minhashemi · 2021-11-16T12:32:33Z

Installation instruction. Pip

Download scikit-learn from this repository

Place it at .wheels directory of your project

Install scipy from Anaconda
pip install scipy --index-url=https://pypi.anaconda.org/scipy-wheels-nightly/simple
Install scikit-learn
pip install .wheels/scikit_learn-1.1.dev0-cp39-cp39-macosx_11_0_universal2.whl

How to put the downloaded package in .wheels directory? I use visual studio code on M1 MacBook Pro MacOS Monterey

MikhailKravets · 2021-11-17T06:13:43Z

You can keep the downloaded package at whatever directory you want. .wheels is just an example. You can place it in the project root then

pip install scikit_learn-1.1.dev0-cp39-cp39-macosx_11_0_universal2.whl

rgommers · 2021-11-17T06:21:55Z

pip install scipy --index-url=https://pypi.anaconda.org/scipy-wheels-nightly/simple

Just a note on these: they may crash your OS if you're on macOS 11. It's recommended to upgrade to macOS 12. SciPy wheels for macOS >=12 should be coming soon: https://mail.python.org/archives/list/scipy-dev@python.org/thread/LLN2O4G2XI2MPILRW2XRRVCUK336WGKF/.

danieltomasz · 2021-11-24T19:25:18Z

Scipy just released 1.7.3 with binary wheels
https://github.com/scipy/scipy/releases/tag/v1.7.3

juceyj · 2021-11-25T12:02:53Z

Installation instruction. Pip

Download scikit-learn from this repository

Place it at .wheels directory of your project

Install scipy from Anaconda
pip install scipy --index-url=https://pypi.anaconda.org/scipy-wheels-nightly/simple
Install scikit-learn
pip install .wheels/scikit_learn-1.1.dev0-cp39-cp39-macosx_11_0_universal2.whl
I finally managed to install it. I have already tried this morning, but probably I messed something. Python 3.9, latest version of pip, and M1 Pro

I already installed the sklearn but it still has somthing wrong.

numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject

alexcombessie · 2022-02-01T23:34:51Z

Hi,

Any updates of this - for folks like me who don't want conda, just pip ?

Cheers,

Alex

sunilshah · 2022-02-02T03:58:49Z

Hi,

Any updates of this - for folks like me who don't want conda, just pip ?

Cheers,

Alex

On Mac OS 12.2, I have been running scikit-learn 1.0.2 in native mode. Scipy 1.7.3 and python 3.9.10 were installed using homebrew. I used pip3 install -U --no-use-pep517 scikit-learn

turnUpTheChill · 2022-03-10T19:32:42Z

FWIW, running on an M1 I found using docker to run a jupyter notebook was the only viable work-around I could use

peterfortuin · 2022-04-26T08:40:54Z

I'm looking forward to use scikit-learn on my M1 Macbook. Keep up the good work!

peartobear · 2022-09-27T07:31:56Z

Any updates on this?

ogrisel added New Feature Build / CI labels Jan 8, 2021

This was referenced Jan 8, 2021

Bus Error 10 on Apple Silicon compiled for python 3.8 #19097

Closed

DOC Update installation instructions for macos/arm64 #19146

Merged

usr-ein mentioned this issue Jan 31, 2021

BLD: fail to build on Apple M1 numpy/numpy#17807

Closed

lutzroeder mentioned this issue Feb 14, 2021

Apple Silicon support lutzroeder/netron#627

Closed

7 tasks

AndrewRPorter mentioned this issue Mar 12, 2021

"No module named '_lightfm_fast_no_openmp'" on a Macbook pro-2017 lyst/lightfm#523

Open

onedrawingperday mentioned this issue Mar 30, 2021

Upstream: Installation broken on MacOS with Apple Silicon Ars-Linguistica/mlconjug3#215

Closed

katkaypettitt mentioned this issue Aug 19, 2021

unable to build/install on MacOS/M1 scipy/scipy#14587

Closed

dmill166 mentioned this issue Nov 7, 2021

Minor Progress dmill166/datamining-learning#31

Merged

thomasjpfan mentioned this issue Nov 30, 2021

CI Build macOS arm wheels [cd build gh] #21827

Merged

ogrisel closed this as completed in #21827 Dec 2, 2021

Craigacp mentioned this issue Dec 23, 2021

Jupyter kernel dies / Segmentation fault : 11, when upgrading xgboost to > v0.90 on MacOS M1 Chip dmlc/xgboost#7504

Closed

SamEdwardes mentioned this issue May 17, 2022

poetry add scikit-learn does not work on m1 mac python-poetry/poetry#5628

Closed

3 tasks

GokuMohandas mentioned this issue Feb 3, 2023

Unable to install dependencies on Mac M1 GokuMohandas/mlops-course#22

Closed

This was referenced Jun 9, 2023

Clean UP & Dockerization of eos74bo ersilia-os/eos74bo#20

Closed

modified scikit-learn to install from conda-forge ersilia-os/eos74bo#23

Merged

betatim mentioned this issue Aug 14, 2023

DOC Remove outdated instructions for Apple Silicon #27067

Merged

Native wheels for the macos/arm64 platform (Apple Silicon M1 hardware) #19137

Native wheels for the macos/arm64 platform (Apple Silicon M1 hardware) #19137

ogrisel commented Jan 8, 2021 •

edited

ogrisel commented Jan 8, 2021

ogrisel commented Feb 19, 2021

UnixJunkie commented Jun 9, 2021

grisaitis commented Jun 22, 2021 •

edited

UnixJunkie commented Jun 23, 2021

jnothman commented Jun 24, 2021 via email

anirudh-g commented Sep 2, 2021

csp197 commented Oct 5, 2021

Mantower commented Oct 6, 2021 •

edited

appleseedexm commented Oct 8, 2021 •

edited

HRezaei commented Oct 17, 2021 •

edited

MikhailKravets commented Oct 18, 2021 •

edited

sunilshah commented Oct 18, 2021 •

edited

nimafanniasl commented Oct 26, 2021 •

edited

danieltomasz commented Oct 26, 2021 •

edited

ShashankKrishnaV commented Oct 27, 2021

SiddyP commented Nov 1, 2021

xiangpeng2008 commented Nov 3, 2021

MarcoTerzulli commented Nov 4, 2021 •

edited

danieltomasz commented Nov 4, 2021

MarcoTerzulli commented Nov 4, 2021

MarcoTerzulli commented Nov 4, 2021 •

edited

Installation instruction. Pip

Karan-S-Mittal commented Nov 8, 2021

matteosantama commented Nov 13, 2021

minhashemi commented Nov 16, 2021

Installation instruction. Pip

MikhailKravets commented Nov 17, 2021

rgommers commented Nov 17, 2021

danieltomasz commented Nov 24, 2021

juceyj commented Nov 25, 2021

Installation instruction. Pip

alexcombessie commented Feb 1, 2022

sunilshah commented Feb 2, 2022

turnUpTheChill commented Mar 10, 2022

peterfortuin commented Apr 26, 2022

peartobear commented Sep 27, 2022

Native wheels for the macos/arm64 platform (Apple Silicon M1 hardware) #19137

Native wheels for the macos/arm64 platform (Apple Silicon M1 hardware) #19137

Comments

ogrisel commented Jan 8, 2021 • edited

ogrisel commented Jan 8, 2021

ogrisel commented Feb 19, 2021

UnixJunkie commented Jun 9, 2021

grisaitis commented Jun 22, 2021 • edited

UnixJunkie commented Jun 23, 2021

jnothman commented Jun 24, 2021 via email

anirudh-g commented Sep 2, 2021

csp197 commented Oct 5, 2021

Mantower commented Oct 6, 2021 • edited

appleseedexm commented Oct 8, 2021 • edited

HRezaei commented Oct 17, 2021 • edited

MikhailKravets commented Oct 18, 2021 • edited

So installation instruction. Pipenv

Installation instruction. Pip

sunilshah commented Oct 18, 2021 • edited

nimafanniasl commented Oct 26, 2021 • edited

danieltomasz commented Oct 26, 2021 • edited

ShashankKrishnaV commented Oct 27, 2021

SiddyP commented Nov 1, 2021

xiangpeng2008 commented Nov 3, 2021

MarcoTerzulli commented Nov 4, 2021 • edited

danieltomasz commented Nov 4, 2021

MarcoTerzulli commented Nov 4, 2021

MarcoTerzulli commented Nov 4, 2021 • edited

Installation instruction. Pip

Karan-S-Mittal commented Nov 8, 2021

matteosantama commented Nov 13, 2021

minhashemi commented Nov 16, 2021

Installation instruction. Pip

MikhailKravets commented Nov 17, 2021

rgommers commented Nov 17, 2021

danieltomasz commented Nov 24, 2021

juceyj commented Nov 25, 2021

Installation instruction. Pip

alexcombessie commented Feb 1, 2022

sunilshah commented Feb 2, 2022

turnUpTheChill commented Mar 10, 2022

peterfortuin commented Apr 26, 2022

peartobear commented Sep 27, 2022

ogrisel commented Jan 8, 2021 •

edited

grisaitis commented Jun 22, 2021 •

edited

Mantower commented Oct 6, 2021 •

edited

appleseedexm commented Oct 8, 2021 •

edited

HRezaei commented Oct 17, 2021 •

edited

MikhailKravets commented Oct 18, 2021 •

edited

sunilshah commented Oct 18, 2021 •

edited

nimafanniasl commented Oct 26, 2021 •

edited

danieltomasz commented Oct 26, 2021 •

edited

MarcoTerzulli commented Nov 4, 2021 •

edited

MarcoTerzulli commented Nov 4, 2021 •

edited