This file gives an overview of what is necessary to build binary releases for NumPy. Windows binaries are built here using Wine, they can of course also be built on Windows itself. Building OS X binaries on another platform is not possible, but our current OSX binary build procedure uses travis-ci virtual machines running OSX.
Current build and release info
The current info on building and releasing NumPy and SciPy is scattered in several places. It should be summarized in one place, updated and where necessary described in more detail. The sections below list all places where useful info can be found.
- http://www.scipy.org/Installing_SciPy and links on that page.
Supported platforms and versions
Python 2.6-2.7 and >=3.2 are the currently supported versions when building from source. We test numpy against all these versions every time we merge code to trunk. Binary installers may be available for a subset of these versions (see below).
Python 2.7 and >=3.3 are the versions for which we provide binary installers. OS X versions >= 10.6 are supported. We build binary wheels for OSX that are compatible with Python.org Python, system Python, homebrew and macports - see this OSX wheel building summary for details.
32-bit Python 2.7, 3.3, 3.4 are the versions for which we provide binary installers. Windows XP, Vista and 7 are supported. Our current windows mingw toolchain is not able to build 64-bit binaries of numpy. We are hoping to update to a mingw-w64 toolchain soon.
Many distributions include NumPy. Building from source is also relatively straightforward. Only tarballs are created for Linux, no specific binary installers are provided (yet).
BSD / Solaris
No binaries are provided, but succesful builds on Solaris and BSD have been reported.
The same gcc version is used as the one with which Python itself is built on each platform. At the moment this means:
- OS X builds on travis currently use clang. It appears that binary wheels for OSX >= 10.6 can be safely built from from OSX 10.9 when building against the Python from the Python.org installers.
- Windows builds use MinGW 3.4.5. Updating this to a more recent MinGW with GCC 4.x is desired, but there are still practical difficulties in building the binary installers.
You will need Cython for building the binaries. Cython compiles the
files in the numpy distribution to
Fortran: on OS X gfortran from this site is used. On Windows g77 (included in MinGW) is the current default, in the future this may shift to gfortran as well.
- Python(s) from python.org
- Texlive (or MikTeX on Windows)
For building Windows binaries on OS X Wine can be used. In Wine the following needs to be installed:
- Python 2.6-2.7 and 3.3
- CpuId plugin for MakeNsis : this can be found in the NumPy source tree under tools/win32build/cpucaps and has to be built with MinGW (see SConstruct file in that dir for details)
- ATLAS, 3x ([No SSE, SSE2, SSE3] for superpack installer) : ATLAS does not compile under wine or on Windows out of the box. Binaries for ATLAS can be found in the vendor repository on GitHub (http://github.com/numpy/vendor).
To install Wine on OS X Snow Leopard the current options are to compile a
current unstable version ,`<http://wiki.winehq.org/MacOSX/Building>`_, or to use
an install script from here. For
me, the former option did not work (everything compiled, but after installing
Python the command
import tempfile resulted in an exception. The latter
option did work.
After successful installation and an invocation of the wine executable, a ~/.wine folder exists - new programs will be installed there in ~/.wine/drive_c. Installing Windows programs with .exe executables is done by running
$ wine yourprog.exe
and MSI installers can be installed with
$ msiexec /i yourprog.msi
For the above to work you probably need to put the wine-1.x.x/bin directory in your PATH.
To install MinGW, the easiest option is to use the automated installer on the MinGW download page. This will give you (at this moment) GCC 3.4.5; GCC 4.x is still not supported officially by MinGW.
To be able to use gcc and MakeNsis in Wine, the locations of gcc.exe and makensis.exe should be added to the Windows environment variable PATH. This can easily be done by running
$ wine regedit
add adding a PATH variable in HKEY_CURRENT_USER/Environment.
Virtualenv is a very useful tool to keep several versions of packages around. It is also used in the Paver script to build the docs.
What is released
Windows binary installers in "superpack" form for Python 2.7/3.3/3.4. A superpack contains three builds, for SSE2, SSE3 and no SSE.
OSX wheels built via travis-ci : see - see `building OSX wheels`_.
- Release Notes
We build source releases in both .zip and .tar.gz formats.
Agree on a release schedule
A typical release schedule is one beta, two release candidates and a final release. It's best to discuss the timing on the mailing list first, in order for people to get their commits in on time, get doc wiki edits merged, etc. After a date is set, create a new maintenance/x.y.z branch, add new empty release notes for the next version in the master branch and update the Trac Milestones.
Make sure current trunk builds a package correctly
git clean -fxd python setup.py bdist python setup.py sdist
To actually build the binaries after everything is set up correctly, the release.sh script can be used. For details of the build process itself it is best to read the pavement.py script.
The following steps are repeated for the beta(s), release candidates(s) and the final release.
Check that docs can be built
cd doc/ make dist
to check that the documentation is in a buildable state. See doc/HOWTO_BUILD_DOCS.rst.txt for more details and for how to update http://docs.scipy.org.
Before the release branch is made, it should be checked that all deprecated code that should be removed is actually removed, and all new deprecations say in the docstring or deprecation warning at what version the code will be removed.
Check the C API version number
The C API version needs to be tracked in three places
There are three steps to the process.
If the API has changed, increment the C_API_VERSION in setup_common.py. The API is unchanged only if any code compiled against the current API will be backward compatible with the last released NumPy version. Any changes to C structures or additions to the public interface will make the new API not backward compatible.
If the C_API_VERSION in the first step has changed, or if the hash of the API has changed, the cversions.txt file needs to be updated. To check the hash, run the script numpy/core/cversions.py and note the api hash that is printed. If that hash does not match the last hash in numpy/core/code_generators/cversions.txt the hash has changed. Using both the appropriate C_API_VERSION and hash, add a new entry to cversions.txt. If the API version was not changed, but the hash differs, you will need to comment out the previous entry for that API version. For instance, in NumPy 1.9 annotations were added, which changed the hash, but the API was the same as in 1.8. The hash serves as a check for API changes, but it is not definitive.
If steps 1 and 2 are done correctly, compiling the release should not give a warning "API mismatch detect at the beginning of the build.
The numpy/core/include/numpy/numpyconfig.h will need a new NPY_X_Y_API_VERSION macro, where X and Y are the major and minor version numbers of the release. The value given to that macro only needs to be increased from the previous version if some of the functions or macros in the include files were deprecated.
The C ABI version number in numpy/core/setup_common.py should only be updated for a major release.
Check the release notes
Check that the release notes are up-to-date.
Write or update the release notes in a file named for the release, such as
Mention at least the following:
- major new features
- deprecated and removed features
- supported Python versions
- for SciPy, supported NumPy version(s)
- outlook for the near future
Also make sure that as soon as the branch is made, there is a new release notes file in trunk for the next release.
Update the release status and create a release "tag"
Identify the commit hash of the release, e.g. 1b2e1d63ff.
- git co 1b2e1d63ff # gives warning about detached head
First, change/check the following variables in
pavement.py depending on the
RELEASE_NOTES = 'doc/release/1.7.0-notes.rst' LOG_START = 'v1.6.0' LOG_END = 'maintenance/1.7.x'
Do any other changes. When you are ready to release, do the following changes:
diff --git a/setup.py b/setup.py index b1f53e3..8b36dbe 100755 --- a/setup.py +++ b/setup.py @@ -57,7 +57,7 @@ PLATFORMS = ["Windows", "Linux", "Solaris", "Mac OS- MAJOR = 1 MINOR = 7 MICRO = 0 -ISRELEASED = False +ISRELEASED = True VERSION = '%d.%d.%drc1' % (MAJOR, MINOR, MICRO) # Return the git revision as a string
And make sure the
VERSION variable is set properly.
Now you can make the release commit and tag. We recommend you don't push the commit or tag immediately, just in case you need to do more cleanup. We prefer to defer the push of the tag until we're confident this is the exact form of the released code (see: :ref:`push-tag-and-commit`):
git commit -s -m "REL: Release." setup.py git tag -s <version>
-s flag makes a PGP (usually GPG) signed tag. Please do sign the
The release tag should have the release number in the annotation (tag message). Unfortunately the name of a tag can be changed without breaking the signature, the contents of the message cannot.
See : https://github.com/scipy/scipy/issues/4919 for a discussion of signing release tags, and http://keyring.debian.org/creating-key.html for instructions on creating a GPG key if you do not have one.
To make your key more readily identifiable as you, consider sending your key to public keyservers, with a command such as:
gpg --send-keys <yourkeyid>
Apply patch to fix bogus strides
NPY_RELAXED_STRIDE_CHECKING was made the default in Numpy 1.10 and bogus strides are used in the development branch to smoke out problems. The patch should be updated if necessary and applied to the release branch to rationalize the strides.
Update the version of the master branch
Increment the release number in setup.py. Release candidates should have "rc1" (or "rc2", "rcN") appended to the X.Y.Z format.
Also create a new version hash in cversions.txt and a corresponding version define NPY_x_y_API_VERSION in numpyconfig.h
Trigger the OSX builds on travis
See build OSX wheels.
You may need to check the
.travis.yml file of the
Make sure that the releast tag has been pushed, and that the
is set thusly:
- NP_COMMIT=latest-tag # comment out to build version in submodule
Trigger a build by doing an empty (or otherwise) commit to the repository:
cd /path/to/numpy-wheels git commit --allow-empty git push
The wheels, once built, appear in http://wheels.scipy.org
Trigger Windows builds on Appveyor
See: build Windows wheels
- Clone / update the https://github.com/numpy/windows-wheel-builder repository;
- Check the
appveyor.ymlfile in that repository;
- Edit the line starting
NP_VERSION:to give the numpy tag that you want to build;
- Push up to github to trigger a build.
The wheels appear in a Rackspace CDN container at:
The contents via the HTTPS URL seems to get updated more slowly than via the HTTP URL, so if you need the binaries quickly, prefer the HTTP URL.
Trigger Manylinux builds on travis-ci
Until we move the manylinux build scripts, you'll need to ask
@matthew-brett to make you a collaborator on the manylinux repos.
- Clone / update the repository at https://github.com/matthew-brett/manylinux-builds
- Edit the line in
NUMPY_VERSIONS=to set the numpy tag to build;
- Push your edits to
.travis.ymlup to github to trigger a mass manylinux build;
- Clone / update the repository at https://github.com/matthew-brett/manylinux-testing;
- Push an empty commit to the
manylinux-testingrepo to trigger a test run of the newly-built numpy wheels with a range of dependent libraries, as well as numpy's own unit tests. The tests will take several hours.
The built wheels will be available from a Rackspace CDN container at:
As for the other Rackspace containers, the HTTP address may update first, and
you should wait 15 minutes after the build finishes before fetching the
binaries. For the manylinux wheels, the time to run
much greater than 15 minutes, so waiting for the tests to pass will be enough
time for the binaries to refresh on Rackspace.
Make the release
Build the changelog and notes for upload with:
The tar-files and binary releases for distribution should be uploaded to SourceForge, together with the Release Notes and the Changelog. Uploading can be done through a web interface or, more efficiently, through scp/sftp/rsync as described in the SourceForge upload guide. For example:
scp <filename> <username>,firstname.lastname@example.org:/home/frs/project/n/nu/numpy/NumPy/<releasedir>/
The wheels and source should be uploaded to PyPI.
You should upload the wheels first, and the source formats last, to make sure that pip users don't accidentally get a source install when they were expecting a binary wheel.
You can do this automatically using the
wheel-uploader script from
https://github.com/MacPython/terryfy. Here is the recommended incantation for
downloading all the Windows, Manylinux, OSX wheels and uploading to PyPI.
cd ~/wheelhouse # local directory to cache wheel downloads MANYLINUX_URL=http://ccdd0ebb5a931e58c7c5-aae005c4999d7244ac63632f8b80e089.r77.cf2.rackcdn.com WINDOWS_URL=http://58688808cd85529d4031-38dee5dca2544308e91131f21428d924.r12.cf2.rackcdn.com OSX_URL=http://wheels.scipy.org wheel-uploader -u $MANYLINUX_URL -v -s -t manylinux1 numpy 1.11.1rc1 wheel-uploader -u $WINDOWS_URL -v -s -t win numpy 1.11.1rc1 wheel-uploader -u $OSX_URL -v -s -t macosx numpy 1.11.1rc1
-v flag gives verbose feedback,
-s causes the script to sign the
wheels with your GPG key before upload.
You may well find that these uploads break at some point, with error messages from the PyPI server. In this case you'll have to continue the uploads by hand using twine, using something like:
twine upload -s numpy-1.11.1rc1-cp34-*.whl
Do this for the wheel files that
wheel-uploader downloaded, but for which
the upload failed.
warehouse PyPI server seems to be more reliable in receiving automated
wheel uploads. You can set the repository to upload to with the
twine. The warehouse repository URL for your
~/.pypirc file is https://upload.pypi.io/legacy/
There are two ways to update the source release on PyPI, the first one is:
$ git clean -fxd # to be safe $ python setup.py sdist --formats=gztar,zip # to check # python setup.py sdist --formats=gztar,zip upload --sign
This will ask for your key PGP passphrase, in order to sign the built source packages.
The second way is to upload the PKG_INFO file inside the sdist dir in the web interface of PyPI. The source tarball can also be uploaded through this interface.
To push the travis-ci OSX wheels up to PyPI see : https://github.com/MacPython/numpy-wheels#uploading-the-built-wheels-to-pypi
Push the release tag and commit
Finally, now you are confident this tag correctly defines the source code that you released you can push the tag and release commit up to github:
git push # Push release commit git push upstream <version> # Push tag named <version>
upstream points to the main https://github.com/numpy/numpy.git
All documentation for a release can be updated on http://docs.scipy.org/ with:
make dist make upload USERNAME=<yourname> RELEASE=1.11.0
<username> must have SSH credentials on the server. If you don't
have those, ask someone who does (the list currently includes @rgommers,
@juliantaylor and @pv).
Also rebuild and upload
docs.scipy.org front page, if the release
series is a new one. The front page sources have their own repo:
https://github.com/scipy/docs.scipy.org. Do the following:
index.rstfor the new version.
- Check that the built documentation is OK.
make upload USERNAME=<username> RELEASE=1.x.y
A release announcement with a link to the download site should be placed in the sidebar of the front page of scipy.org.
The scipy.org should be a PR at https://github.com/scipy/scipy.org. The file
that needs modification is
www/index.rst. Search for
Announce to the lists
The release should be announced on the mailing lists of NumPy and SciPy, to python-announce, and possibly also those of Matplotlib,IPython and/or Pygame.
During the beta/RC phase an explicit request for testing the binaries with several other libraries (SciPy/Matplotlib/Pygame) should be posted on the mailing list.
Announce to Linux Weekly News
Email the editor of LWN to let them know of the release. Directions at: https://lwn.net/op/FAQ.lwn#contact
After the final release
After the final release is announced, a few administrative tasks are left to be done:
- Forward port changes in the release branch to release notes and release scripts, if any, to trunk.
- Update the Milestones in Trac.