enhancements distutils_preprocessing

robertwb edited this page Dec 14, 2010 · 11 revisions

CEP 201 - Distutils Preprocessing

  • Status: Released as part of 0.14 (December 2010)


Distutils is the standard way to build Cython extension modules for large projects. Typically one invokes setup(...) which then indirectly invokes Cython when a .pyx source file is encountered. This limits the control we have over the pyx to c compilation process. This proposal is to make a standard Cython method that "preparses" the list of Extension modules and converts all .pyx source files to .c(pp) ones before passing them off to distutils. Doing this step ourselves, all at once, would have several advantages including:

  • Cython-aware dependancy tracking (Done)
  • in-file directives for language, libraries, compiler flags, etc. (which could be transitive) (Done)
  • efficient multi-file compilation (especially with shared deep cimports) (Done)
  • sharing common utility code in a single .h file
  • shared cimported type declaration code
  • whole-program type inference or other analysis

This will resolve other nuisances such as the difficulty in supporting parallel Pyrex and Cython installs, or relying on distutils to know about Cython at all. Disentangling the Cythonization phase from the compile process and using in-file directives will allow easier support for other build systems as well.


Rather than writing a setup.py that looks like

from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext

    Extension("primes",       ["primes.pyx"]),
    Extension("spam",         ["spam.pyx"]),

  name = 'MyProject',
  cmdclass = {'build_ext': build_ext},
  ext_modules = ext_modules,

one would write

from distutils.core import setup
from Cython.Build import cythonize

  name = 'MyProject',
  ext_modules = cythonize(["*.pyx"]),

The cythonize command takes a list of filenames/filename patterns and returns a list of Extension classes ready to be compiled by distutils, using timestamps and its understanding of Cython's include and cimport statements to compile all and only the necessary extensions. It is also permissible to do

    Extension("primes", ["primes.pyx"]),
    Extension("spam",   ["spam.pyx"], libraries = ["spam"]),

  name = 'MyProject',
  ext_modules = cythonize(ext_modules),

however the preferred way to specify distutils options is directly in the .pyx or .pxd file itself, via a comment similar to how module-level cython directives are given. For example, one can write

# distutils: language = c++
# distutils: libraries = spam eggs
# distutils: include_dirs = /opt/food/include

When such declarations appear in .pxd files, they are transitive (if it make sense), so modules cimporting from them will automatically propagate the right compile flags to use them as well.

The cythonize command takes several options, including an nthreads parameter to use multiprocessing to compile .pyx files in parallel. See http://hg.cython.org/cython-devel/file/5a033978dd11/Cython/Build/Dependencies.py#l442 .


  • One still may need to read and understand sys.argv to know whether or not to invoke Cython.
Clone this wiki locally
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.