CythonExtensionsOnWindows

Compiling C extension modules on Windows

This page describes how to compile a Cython extension (works for CPython extensions also) for Windows.

If you encounter one of the following errors:

$ python setup.py build_ext --inplace --compiler=msvc
running build_ext
building 'helloworld' extension
error: Unable to find vcvarsall.bat
$ python setup.py build_ext --inplace --compiler=msvc
collect2: ld returned 1 exit status
error: command 'c:\\mingw-gcc\\bin\\gcc.exe' failed with exit status 1

Then you are at the right place! These errors simply mean that you lack a C compiler to compile the .c files generated by Cython from your .pyx files. On GNU/Linux, the solution is easy, and most users do not even have to install anything special because the platform compiler gcc is already installed. On Windows, it is a little bit more complicated, since the platform compiler is not installed by default, so you have to do it by yourself.

Below we describe the two currently available ways to install the platform compiler on Windows and set it up so that you can compile C extension for Python on Windows.

Using Microsoft Visual C++ Compiler for Python (only for Python 2.7.x)

NOTE: the following works only for Python 2.7 and 3.2, for Python 3.3 and 3.4 you should try the older method with the SDK.

The easiest way to compile correctly working C and Cython extensions for Python on Windows is to use Visual Studio SDK, because incompatibilities may arise when using other compilers. However, the process (outlined below) can be quite daunting and will consume several GB of install, only to get a working compiler.

Microsoft fairly recently released a freely distributed version of their compiler specifically for building Python extensions for the python.org 2.7 series, called Microsoft Visual C++ Compiler for Python, which is only ~85MB and should work for most Python compiling purposes.

This version does not have a couple of bugs that made it hard to get set up properly for 64 bit extensions. Before proceeding to the instructions below, you should update setuptools to at least 6.0.

Here is how to compile C extensions using Microsoft Visual C++ Compiler for Python:

Install Microsoft Visual C++ Compiler for Python from http://www.microsoft.com/en-us/download/details.aspx?id=44266

Create a setup.py loading setuptools instead of distutils. setuptools will monkey-patch distutils and find vcvarsall.bat. For example, this won't work:

from distutils.core import setup
from distutils.extension import Extension

But the following will:

try:
    from setuptools import setup
    from setuptools import Extension
except ImportError:
    from distutils.core import setup
    from distutils.extension import Extension

For the rest of setup.py, read the tutorial.

When using a IPython Notebook, you can apply the same trick:

#First cell:
    import setuptools
    %load_ext Cython

#Second cell:
    %%cython -a
    import cython
    cimport cython

    cdef int alpha = 255
    print alpha

Now, you should be able to compile your C extensions, and even Cython extensions if you have Cython installed. If you still get an error about not finding vcvarsall.bat, you may try the following workarounds:

Install Microsoft Visual C++ Compiler for Python (as described above).
Launch MSVC for Python command prompt (go to start menu > Microsoft Visual C++ Compiler Package for Python 2.7 > Visual C++ 2008 32-bit Command Prompt)
Enter the following commands:
```
SET DISTUTILS_USE_SDK=1
SET MSSdk=1
```
You can then cd X:\your\path to navigate to your Python app and then build your C extensions by entering:
```
python.exe setup.py build_ext --inplace --compiler=msvc
```

Steps 2 and 3 must be reproduced everytime before building your C extensions. This is because of an issue with the VCForPython27.msi which installs the header files and vcvarsall.bat in folders with a different path layout than the VS2008 SDK and thus confuses the compiler detection of distutils. This will maybe get fixed in distutils for Python 2.7.11.

However, if you want to use IPython/Jupyter and the %%cython magic, you cannot specify the --compiler nor enter the required variables to workaround the detection issue (thus %%cython will always try to compile using mingw).

Another way to workaround this issue, without needing to set variables everytime, is to patch the file C:\yourpythoninstall\Lib\distutils\msvc9compiler.py by adding the following code at the top of the find_vcvarsall() function:

def find_vcvarsall(version):
    """Find the vcvarsall.bat file

    At first it tries to find the productdir of VS 2008 in the registry. If
    that fails it falls back to the VS90COMNTOOLS env var.
    """
    vsbase = VS_BASE % version
    vcpath = os.environ['ProgramFiles']
    vcpath = os.path.join(vcpath, 'Common Files', 'Microsoft',
        'Visual C++ for Python', '9.0', 'vcvarsall.bat')
    if os.path.isfile(vcpath): return vcpath
    vcpath = os.path.join(os.environ['LOCALAPPDATA'], 'Programs', 'Common',
                          'Microsoft', 'Visual C++ for Python', '9.0',
                          'vcvarsall.bat')
    if os.path.isfile(vcpath): return vcpath
    ...

Then, create a file distutils.cfg in the same folder, and put this inside:

[build]
compiler=msvc

Now you can compile C extensions without any prior manipulations. Thus you can now call the %%cython magic inside IPython/Jupyter, by placing this inside a cell:

%load_ext Cython
%%cython

For more information about this detection issue, see the bug report and workaround by Gregory Szorc: http://bugs.python.org/issue23246

And for a discussion about providing more info on compiler detection errors: https://github.com/pypa/pip/issues/942, http://bugs.python.org/issue2943

Note: Enthought Canopy users, see here

Note2: don't try the GccWithBinaries project, even through it will install with a 32-bit Python (whatever the bits of the installed Windows), Cython extension won't compile. Same with standard MinGW, and same with GCC. https://github.com/develersrl/gccwinbinaries

Note3: to compile your Cython scripts, you need to either use pyximport, or make a setup.py as described at <http://docs.cython.org/src/tutorial/cython_tutorial.html> and <http://docs.cython.org/src/reference/compilation.html>.

You can also try distcontrib <http://distcontrib.readthedocs.org/en/latest/> which seems to generate setup.py compatible with Cython.

Using Windows SDK C/C++ compiler (works for all Python versions)

The standard way to compile C extensions for CPython (including Cython extensions) is to install the compiler that was used to compile the CPython interpreter. CPython is always compiled using Visual Studio.

In order to do this, you either need the Visual Studio that compiled CPython or a Windows SDK C/C++ compiler. Now the problem is that depending on which Python version you want to compile your extension for, you need a different version of the SDK, because they are not cross-compatible.

Luckily, people have compiled the list of CPython versions with their corresponding SDK, and in fact, you can use the free Visual Studio Express v9.0 package to compile Python between versions 2.6.9 - 3.3.4, VSE v10.0 (2010) to compile Python 3.3.5 - 3.4.x, and Visual Studio 2015 Community Edition for Python 3.5. Make sure that you check the "Common tools for Visual C++" option during the Visual Studio install (in order to avoid the dreaded vcvarsall.bat error). Note that this only works for compiling 32-bits extensions, for 64-bits extensions you will need to install in addition the corresponding Windows SDK below.

Here are two good listings where you will find all the necessary infos to install the correct Visual Studio for your Python:

If you don't want to install Visual Studio, you can choose to install the Windows SDK C/C++ compiler, as indicated in the table below:

Python	Windows SDK / .NET	Download the GRMSDKX_EN_DVD.iso
2.6 2.7 3.1	MS Windows SDK for Windows 7 and .NET Framework 3.5 SP1	Windows SDK (.NET 3.5 SP1)
3.3	MS Windows SDK for Windows 7 and .NET Framework 4	Windows SDK (.NET 4)

At this point it does not matter if you want to compile for 64 or 32 bit. The version you downloaded can do both.

Before you run something along the lines of :

pip install package_name

or :

python setup.py command_name

to install or build a package, you do have to start a SDK Command Prompt or CMD Shell and set some environment variables.

By default the shell starts in at the installation path of the Windows SDK (C:\Program Files\Microsoft SDKs\Windows\v7.0). There, we have to to two things:

Tell distutils / setuptools to use the Microsoft SDK compiler
Tell the compiler to compile a 32 or 64 bit release and whether it should be a debug or a release build

Thus, we have to enter two commands :

set DISTUTILS_USE_SDK=1
setenv /x64 /release

Assuming you haven't changed the Windows SDK installation path, this looks something like this :

C:\Program Files\Microsoft SDKs\Windows\v7.0>set DISTUTILS_USE_SDK=1
C:\Program Files\Microsoft SDKs\Windows\v7.0>setenv /x64 /release

As you might have guessed, you can swap out x64 for x86 or /release for /debug and vice versa.

Then, switch to your project directory and try to python setup.py command_name or pip install package_name. This should work now.

Finally, you should distribute your binary package as a .whl file to PyPi.

Additional information

The C/C++ compiler and linker that comes with the Windows SDK should be used because Python itself is compiled with it. That way your extensions will always be binary compatible with Python and other installed system components and libraries.

You should find the SDK Command Prompt or the CMD Shell it in the start menu. In case you don't find it, check your Windows SDK installation path (default: C:\Program Files\Microsoft SDKs\Windows\v7.0\).

Less useful information

Do not use MinGW-w64. As you will notice, the MinGW import library for Python (e.g. libpython27.a) is omitted from the AMD64 version of Python. This is deliberate. Do not try to make one using dlltool. There is no official MinGW-w64 release yet, it is still in "beta" and considered unstable, although you can get a 64-bit build from e.g. TDM-GCC. There have also been issues with the mingw runtime conflicting with the MSVC runtime; this can happen from places you don't expect, such as inside runtime libraries for g++ or gfortran. To stay on the safe side, avoid MinGW-w64 for now.
Those with experience from Linux or Apple MacOSX might pick GCC (MinGW) as their C/C++ compiler of choice on Windows. Note that GCC is not the platform C compiler on Windows. To be binary compatible with most libraries and applications on Windows, use Microsoft's compiler. This is no different from selecting GCC as the compiler of choice on Linux. Microsoft's platform C/C++ compiler for Windows is free of charge.
For C++, observe that GCC (Mingw) and MSVC use different C++ runtimes. If you use C++ in your extension, the C++ runtime must also be compatible with Python's C runtime. For this reason, always use Microsoft's compiler for C++, even on 32-bit x86. This is very important on a platform like Windows where C++ is pervasive (cf. COM and ActiveX). If you use GCC as C++ compiler your extension will not be binary compatible with other C++ libraries or COM, not be compatible with Python's C runtime, and differences in exception handling between GCC and MSVC can crash your application.
If you need Fortran (e.g. for f2py or fwrap), Intel, Absoft and Portland have compilers compatible with Microsoft's C compiler and linker. Be very careful if you use an unofficial MinGW-w64 build of gfortran, as libgfortran is linked against an incompatible C runtime. Unlike C and C++, there is no freely available Fortran compiler except MinGW-w64 gfortran on Windows 64.
If you prefer to use Microsoft Visual C++ for editing and debugging, the SDK can be registered with the commercial version of Microsoft Visual Studio. For Python 2.6, 2.7 and 2.8, you will need Visual Studio 2008.

Provide feedback

Saved searches