Spinning wheels
Wheels are the new standard binary installation format for Python.
Many projects provide wheels now : http://pythonwheels.com
However, only some of these wheels are binary wheels.
In the old days, pip would only install binary wheels on Windows. But, on January 1st 2014, pip started installing matching binary wheels for macOS (the relevant pull request).
Some popular projects that already have such wheels are:
- pyzmq
- pymongo
- billiard
- gnureadline
- numpy
- scipy
- matplotlib
- pandas
- scikit-image
- scikit-learn
- Pillow
- h5py
- Cython
"Matching binary wheels" are wheels with a filename that matches the installing pip's Python version, Python ABI version, and the platform tag.
At first, some worried that this tag system might cause trouble for people installing macOS binary wheels into unusual Python versions, such as homebrew and macports.
In practice though, the platform tag does a good job of preventing pip from installing a wheel that won't work.
Here's an example wheel filename:
numpy-1.8.0-cp27-none-macosx_10_6_intel.whl. The name splits at dashes
into:
- numpy (package name)
- 1.8.0 (package version)
- cp27 (CPython 2.7 - Python version)
- none (Python ABI number - only applies to Pythons >= 3)
- macosx_10_6_intel (platform tag)
The platform tag comes from the output of python -c "import distutils.util; print(distutils.util.get_platform())" on the platform building the wheel.
I (MB) built
numpy-1.8.0-cp27-none-macosx_10_6_intel.whl with Python 2.7 from a
Python.org binary installer on an macOS 10.9 machine. The macOS platform tag
further breaks down to:
- macosx
- 10_6 (the version of the SDK used to compile Python)
- intel (short-hand for a fat binary containing x86_64 and i386 objects)
As you see, the SDK part of the tag is not from the macOS running on the machine
I (MB) was building from but from the distutils configuration on the Python I
was building for. In this case I was building for a Python.org binary, and
that Python.org binary gave macosx-10.6-intel from distutils
get_platform().
For pip to accept the wheel as matching its own platform, the platform value
has to be compatible with your version of Python and macOS. So, pip will
accept wheels specifying 10_6_intel for any version of macOS >= 10.6, with
any of i386 or x86_64 or dual-architecture Python. Pip will accept a wheel
with platform tags 10_12_x86_64 only on macOS >= 10.12, on either
dual-architecture or single-architecture x86_64 Python.
The table below has some values of distutils.util.get_platform() for
different Pythons on macOS.
Read 10.x below as referring to the macOS version you are currently running.
The real value will be 10.12 or similar.
Up until Python 3.7, Python.org installers were built to be compapatible with macOS 10.6, and were dual architecture (i386 and x86_64 == 32- and 64- bit). Python.org Python 3.7 has an alternative installer compatible with Python 10.9 and above, and only containing x86_64 / 64 bit architecture.
| Python source | Python version | macOS version | get_platform() |
|---|---|---|---|
| Python.org | 3.7 | 10.6+ or 10.9+ | macosx-10.9-intel or macosx-10.6-intel (64 bit only on 10.9) |
| Python.org | 2.7 ... 3.6 | 10.6+ | macosx-10.6-intel |
| System Python | 2.7 | 10.x | macosx-10.x-intel (always matches) |
| Macports | 2.7 ...3.7 | 10.x | macosx-10.x-x86_64 (matches latest install) |
| Homebrew | 2.7 ... 3.7 | 10.x | macosx-10.x-x86_64 (matches latest install) |
| Anaconda | 2.7 ... 3.7 | 10.9+ | macosx-10.7-x86_64 |
You get the idea. Most Python.org Pythons use the 10.6 SDK, and have fat (x86_64 and i386) architecture in them; since 3.7 Python.org also provides a 10.9 SDK and x86_64 only version. System Pythons use the SDK for the macOS they ship with, and also have fat (dual) architecture. Homebrew and Macports Python have the SDK for the macOS they were first installed / last updated on, and x86_64 architecture only. Anaconda uses a uniform minimum (currently reported as 10.7, even though they claim 10.9+ support only).
This tells us that wheels built with 10.6 / fat architecture Python.org Python
will have the correct architecture and compatible SDK for all the other
Pythons listed. Why? Because having a fat binary includes having x86_64, so
is compatible with x86_64-only builds. Stuff compiled with the 10.6 SDK
should also be compatible with stuff built against later SDK versions (up to
and including 10.9). You can demonstrate this to yourself by renaming the
wheel above to - for example - numpy-1.8.0-cp27-none-macosx_10_9_x86_64.whl
and then installing into a homebrew python on macOS 10.9. Sure enough, it
installs, imports and tests without problem.
Warning: Apple is removing support for stdlibc++ after 5 years of depreciation; this means that packages with C++ in them that are built with anything older than a 10.9 target may not work on the most recent macOS versions, like 10.14.
Python.org wheels are safe to distribute because the architecture and SDK versions are in fact compatible with system Python, homebrew Python and macports Python.
Strange to say, things can go wrong with wheels as for any binary distribution. Here are some things that can go wrong, and how to fix them:
All Python extensions link against macOS system libraries, but these are carefully managed to be ABI compatible between macOS versions, and you should not run into problems with these.
You can use the delocate utility
to check which libraries you are linking against. For example, this is the
result of running delocate-listdeps --all on a binary wheel for the tornado library:
/usr/lib/libSystem.B.dylib
This library is present and ABI compatible for all of macOS versions 10.6 and higher.
If you build a complicated Python extension it may link against some external
libraries elsewhere on the system. scipy is
one example; it links to the gfortran runtime libraries, whereever it finds
them. Here's the output of delocate-listdeps --all for a scipy wheel
built naively on a standard macOS 10.9 system using gfortran from homebrew:
/System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate
/usr/lib/libSystem.B.dylib
/usr/lib/libstdc++.6.dylib
/usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libgcc_s.1.dylib
/usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libgfortran.3.dylib
/usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libquadmath.0.dylib
Again, the libraries in /System and /usr/lib will be present on macOS >=
10.6, but of course the libraries in /usr/local/Cellar/gfortran will only be
present if someone has installed gfortran via homebrew. If I distribute this
wheel, it will only work for someone who has installed these libraries. The
delocate utility can usually fix this by copying the dynamic libraries into
the wheel and relinking the extensions.
Some people have reported that binaries built with an earlier SDK (such as 10.3) on a later macOS (such as 10.9) do not in fact work on earlier versions of macOS, as they should (see comment on pip PR). I have not myself (MB) run into this problem with the 10.6 SDK. For safety, it is best to build binaries such as wheels on the same macOS versions as the SDK. For example, if you are building wheels targeting the 10.6 SDK, try and build the wheels on a machine running 10.6. I don't know of any reports of problems using these binaries on later macOS versions.
There was some worry on a pip pull-request discussion that it might be possible to get Python confused with wheels built against different C++ runtime libraries. Min RK couldn't make this problem happen with test-cases, so we are currently working on the assumption that this is not an issue. Obviously it doesn't come up if you're not using C++.
If in doubt - test. For example, put your wheels up on a server somewhere (examples of this are Min RKs machine, the nipy server) and then test the wheels with something like:
NIPY_URL=https://nipy.bic.berkeley.edu/scipy_installers
pip install --find-links $NIPY_URL tornado
Do this in virtualenvs with different Pythons and on different macOS versions. If you run into trouble, let us know via the Python Mac special interest group mailing list and we'll try to help. At very least, we'd really like to know.
As we've seen, the Python.org Python distributions are the best to build against, because they use the 10.6 SDK (and hence are compatible with macOS versions from 10.6) and they have dual architectures (i386 and x86_64). This makes the resulting wheel compatible with system Python, homebrew and macports.
Install:
- Python.org Python 2.7
- Python.org Python 3.6
- Dual-architecture Python.org Python 3.7
$MACPIES/2.7/bin/pip install wheel
$MACPIES/3.6/bin/pip3 install wheel
$MACPIES/3.7/bin/pip3 install wheel
Here I'm building wheels for markupsafe:
cd markupsafe
rm -rf build
$MACPIES/2.7/bin/python setup.py bdist_wheel
rm -rf build
$MACPIES/3.6/bin/python3 setup.py bdist_wheel
rm -rf build
$MACPIES/3.7/bin/python3 setup.py bdist_wheel
You should now have three wheels in your distribution directory (usually
dist). In my case:
dist/MarkupSafe-0.23-cp27-none-macosx_10_6_intel.whl
dist/MarkupSafe-0.23-cp36-cp36m-macosx_10_6_intel.whl
dist/MarkupSafe-0.23-cp37-cp37m-macosx_10_6_intel.whl
pip install delocate
delocate-listdeps dist/*.whl
Markupsafe wheels have no dependencies outside the system library paths, so you get something like this:
dist/MarkupSafe-0.23-cp27-none-macosx_10_6_intel.whl:
dist/MarkupSafe-0.23-cp36-cp36m-macosx_10_6_intel.whl:
dist/MarkupSafe-0.23-cp37-cp37m-macosx_10_6_intel.whl:
If your project does have some dependencies from the analysis above, then:
mkdir fixed_wheels
delocate-wheel -w fixed_wheels dist/*.whl
Finally, you can upload these to pypi, maybe using twine. Then you will go green here: http://pythonwheels.com/
We (the MacPython organization) support some other projects building macOS wheels using travis-ci.org; feel free to contact us if you'd like help too.
See Wheel building for details.
At time of writing this paragraph, macOS 10.6 was long since end of life, and there are a very small number of installations in the wild as old as macOS 10.9.
It can be tiresome to build wheels with the very old 10.6 SDK, including the problems with C++ noted above. It is often annoying to build wheels with 32-bit as well as 64-bit architecture.
As a result, it has become fairly common for people to build wheels with the 10.9 SDK. You can do this by setting:
export MACOSX_DEPLOYMENT_TARGET=10.9
before you do your build. Since wheel 0.34.0, this will produce a wheel whose platform tag matches either this macOS version, or the minimum of any libraries that your wheel pulls in, whichever is later.