This file gives an overview of what is necessary to build binary releases for NumPy on OS X. Windows binaries are built here using Wine, they can of course also be built on Windows itself. Building OS X binaries on another platform is not possible.
The current info on building and releasing NumPy and SciPy is scattered in several places. It should be summarized in one place, updated and where necessary described in more detail. The sections below list all places where useful info can be found.
- INSTALL.txt
- release.sh
- pavement.py
- https://github.com/numpy/numpy/blob/master/doc/HOWTO_RELEASE.rst.txt
- http://projects.scipy.org/numpy/wiki/MicrosoftToolchainSupport
- http://www.scipy.org/Installing_SciPy and links on that page.
- http://new.scipy.org/building/windows.html
Python 2.6-2.7 and >=3.2 are the currently supported versions on all platforms.
OS X versions >= 10.5 are supported. Note that there are currently still issues with compiling on 10.7, due to Apple moving to gcc-llvm. Only the Python from python.org is supported. Binaries do not support Apple Python.
Windows XP, Vista and 7 are supported.
Many distributions include NumPy. Building from source is also relatively straightforward. Only tarballs are created for Linux, no specific binary installers are provided (yet).
No binaries are provided, but succesful builds on Solaris and BSD have been reported.
The same gcc version is used as the one with which Python itself is built on each platform. At the moment this means:
- OS X uses gcc-4.0 (since that is what Python itself is built with) up to Python 2.6. Python 2.7 comes in two flavors; the 32-bit version is built with gcc-4.0 and the 64-bit version with gcc-4.2. The "release.sh" script sets environment variables to pick the right compiler. All binaries should be built on OS X 10.5, with the exception of the 64-bit Python 2.7 one which should be built on 10.6.
- Windows builds use MinGW 3.4.5. Updating this to a more recent MinGW with GCC 4.x is desired, but there are still practical difficulties in building the binary installers.
Cython is not needed for building the binaries, because generated C files from Cython sources are checked in at the moment. It is worth keeping an eye on what Cython versions have been used to generate all current C files, it should be the same and most recent version (0.16 as of now).
Fortran: on OS X gfortran from this site is used. On Windows g77 (included in MinGW) is the current default, in the future this may shift to gfortran as well.
- Python(s) from python.org
- virtualenv
- paver
- bdist_mpkg from https://github.com/rgommers/bdist_mpkg (has a necessary patch, don't use the unsupported version on PyPi).
Python itself should be installed multiple times - each version a binary is built for should be installed. The other dependencies only have to be installed for the default Python version on the system. The same applies to the doc-build dependencies below.
- Sphinx
- numpydoc
- Matplotlib
- Texlive (or MikTeX on Windows)
For building Windows binaries on OS X Wine can be used. In Wine the following needs to be installed:
- Python 2.6-2.7 and 3.2
- MakeNsis
- CpuId plugin for MakeNsis : this can be found in the NumPy source tree under tools/win32build/cpucaps and has to be built with MinGW (see SConstruct file in that dir for details)
- MinGW
- ATLAS, 3x ([No SSE, SSE2, SSE3] for superpack installer) : ATLAS does not compile under wine or on Windows out of the box. Binaries for ATLAS can be found in the vendor repository on GitHub (http://github.com/numpy/vendor).
To install Wine on OS X Snow Leopard the current options are to compile a current unstable version ,http://wiki.winehq.org/MacOSX/Building, or to use an install script from here. For me, the former option did not work (everything compiled, but after installing Python the command import tempfile resulted in an exception. The latter option did work.
After successful installation and an invocation of the wine executable, a ~/.wine folder exists - new programs will be installed there in ~/.wine/drive_c. Installing Windows programs with .exe executables is done by running
$ wine yourprog.exe
and MSI installers can be installed with
$ msiexec /i yourprog.msi
For the above to work you probably need to put the wine-1.x.x/bin directory in your PATH.
To install MinGW, the easiest option is to use the automated installer on the MinGW download page. This will give you (at this moment) GCC 3.4.5; GCC 4.x is still not supported officially by MinGW.
To be able to use gcc and MakeNsis in Wine, the locations of gcc.exe and makensis.exe should be added to the Windows environment variable PATH. This can easily be done by running
$ wine regedit
add adding a PATH variable in HKEY_CURRENT_USER/Environment.
Virtualenv is a very useful tool to keep several versions of packages around. It is also used in the Paver script to build the docs.
Windows binaries in "superpack" form for Python 2.6/2.7/3.2/3.3. A superpack contains three builds, for SSE2, SSE3 and no SSE.
OS X binaries are made in dmg format, targeting only the Python from python.org
- Release Notes
- Changelog
A source release in both .zip and .tar.gz formats is released.
A typical release schedule is one beta, two release candidates and a final release. It's best to discuss the timing on the mailing list first, in order for people to get their commits in on time, get doc wiki edits merged, etc. After a date is set, create a new maintenance/x.y.z branch, add new empty release notes for the next version in the master branch and update the Trac Milestones.
The default behavior of the test suite in the master branch is to report errors for DeprecationWarnings and RuntimeWarnings that are issued. For a released version this is not desired. Therefore any known warnings should be solved or explicitly silenced before making the release branch, then when the branch is made, the default behavior should be switched to not raise errors. This is done in the constructor of the NoseTester class in numpy/testing/nosetester.py, by replacing raise_warnings="develop" with raise_warnings="release".
python setup.py bdist
python setup.py sdistTo actually build the binaries after everything is set up correctly, the release.sh script can be used. For details of the build process itself it is best to read the pavement.py script.
Note
The following steps are repeated for the beta(s), release candidates(s) and the final release.
The edits in the documentation wiki suitable for merging should be merged, ideally just before making the release branch. How to do this is described in detail in doc/HOWTO_MERGE_WIKI_DOCS.txt.
Do:
cd doc/
make distto check that the documentation is in a buildable state.
Before the release branch is made, it should be checked that all deprecated code that should be removed is actually removed, and all new deprecations say in the docstring or deprecation warning at what version the code will be removed.
The C API version needs to be tracked in three places
- numpy/core/setup_common.py
- numpy/core/code_generators/cversions.txt
- numpy/core/include/numpy/numpyconfig.h
There are three steps to the process.
- If the API has changed, increment the C_API_VERSION in setup_common.py. The API is unchanged only if any code compiled against the current API will be backward compatible with the last released NumPy version. Any changes to C structures or additions to the public interface will make the new API not backward compatible.
If the C_API_VERSION in the first step has changed, or if the hash of the API has changed, the cversions.txt file needs to be updated. To check the hash, run the script numpy/core/cversions.py and note the api hash that is printed. If that hash does not match the last hash in cversions.txt the hash has changed. Using both the appropriate C_API_VERSION and hash, add a new entry to numpy/core/code_generators/cversions.txt. If the API version was not changed, but the hash differs, you will need to comment out the previous entry for that API version. For instance, in NumPy 1.9 annotations were added, which changed the hash, but the API was the same as in 1.8. The hash serves as a check for API changes, but it is not definitive.
If steps 1 and 2 are done correctly, compiling the release should not give a warning "API mismatch detect at the beginning of the build.
- The numpy/core/include/numpy/numpyconfig.h will need a new NPY_X_Y_API_VERSION macro, where X and Y are the major and minor version numbers of the release. The value given to that macro only needs to be increased from the previous version if some of the functions or macros in the include files were deprecated.
The C ABI version number in numpy/core/setup_common.py should only be updated for a major release.
Check that the release notes are up-to-date, and mention at least the following:
- major new features
- deprecated and removed features
- supported Python versions
- for SciPy, supported NumPy version(s)
- outlook for the near future
Also make sure that as soon as the branch is made, there is a new release notes file in trunk for the next release.
Identify the commit hash of the release, e.g. 1b2e1d63ff.
- ::
git co 1b2e1d63ff # gives warning about detached head
Now, set release=True in setup.py, then
git commit -m "REL: Release." setup.py
git tag <version>
git push origin <version>Increment the release number in setup.py. Release candidates should have "rc1" (or "rc2", "rcN") appended to the X.Y.Z format.
Also create a new version hash in cversions.txt and a corresponding version define NPY_x_y_API_VERSION in numpyconfig.h
The tar-files and binary releases for distribution should be uploaded to SourceForge, together with the Release Notes and the Changelog. Uploading can be done through a web interface or, more efficiently, through scp/sftp/rsync as described in the SourceForge upload guide. For example:
scp <filename> <username>,numpy@frs.sourceforge.net:/home/frs/project/n/nu/numpy/NumPy/<releasedir>/The final release (not betas or release candidates) should be uploaded to PyPi. There are two ways to update PyPi, the first one is:
$ python setup.py sdist uploadand the second one is to upload the PKG_INFO file inside the sdist dir in the web interface of PyPi. The source tarball can also be uploaded through this interface. A simple binary installer for windows, created with bdist_wininst, should also be uploaded to PyPi so easy_install numpy works.
Do the following (or ask the doc people to take care of it):
Rebuild and upload documentation:
cd numpy/docmake dist- Check that the built documentation is OK.
touch output-is-finemake upload UPLOAD_TARGET=USERNAME@docs.scipy.org:/home/docserver/www-root/doc/numpy-1.5.x/
where USERNAME should be replaced by your account on docs.scipy.org, and numpy-1.5.x by the version number of the release series. For instance, for Numpy 1.5.1, it should be numpy-1.5.x and for Numpy 2.0.0 numpy-2.0.x.
Rebuild and upload docs.scipy.org front page, if the release series is a new one. The front page sources are located in the Scipy repository:
cd scipy/doc/frontpage- Edit
_templates/indexcontent.htmlto add links to the new release series. make dist- Check that the built documentation is OK.
touch output-is-finemake upload USER=USERNAME
A release announcement with a link to the download site should be placed in the sidebar of the front page of scipy.org.
The release should be announced on the mailing lists of NumPy and SciPy, to python-announce, and possibly also those of Matplotlib,IPython and/or Pygame.
During the beta/RC phase an explicit request for testing the binaries with several other libraries (SciPy/Matplotlib/Pygame) should be posted on the mailing list.
After the final release is announced, a few administrative tasks are left to be done:
- Forward port changes in the release branch to release notes and release scripts, if any, to trunk.
- Update the Milestones in Trac.