enhancements build

robertwb edited this page Jan 29, 2010 · 4 revisions
Clone this wiki locally

Build System

--Much of this is taken care of in the experimental build system for Sage I am working on that is licensed under BSD. It is specifically designed to be a general build system so that it can be used by other cython projects
--Gary Furnish

-- The above is over two years old, and Gary Furnish is no longer (as far as I know) involved in Sage or Cython.


The question often comes up, why pxd files? The main reason is so that other pyx files can build on dependancies without having to parse lots of other (perhaps unavailable) pyx files. Also, pyx files can be changed without having to worry about recompiling the stuff that builds on top of it.

However, there is a lot of redundancy between pyx and pxd files that could perhaps be eliminated, or at least provide a .pyx -> .pxd file conversation tool. I see many issues with parallel builds here. Perhaps this should be a seperate tool that can precompile pyx files into pickled pxd files. -- Gary


Currently in Sage, each and every file that depends on parent.pxd or element.pxd or cdefs.pxi (i.e. nearly all of them) has to reparse the entire file. Parsing is one of the most significant factors in Cython's runtime. When one is building the Sage library, and going up the same tree multiple times, one has almost O(n^2) runtime for what should be an O(n) operation.

It would be nice if there were a way to cache a parsed pxd file as a pickled object, and read that until the pxd changed.

Perhaps even the output of the file (i.e. all the pre-declarations and declarations of structs, vtabs, etc) could be cached too. Or maybe this should be put in a .h file that just gets included everywhere else?

Precompiled pxd's would be a great addition. Caching the output will break enhancements I'm working on that solves certain issues with code generation order that allows very limited circular imports. -- Gary


There is no builtin system for checking dependancies in Cython. We have a simple system for doing so in Sage, but it's based mostly on searching for strings in files. This should be part of Cython. (When we first started using Pyrex in Sage the solution, when one got weird TypeErrors at runtime, was to touch random .pyx files until it worked, or, if all else failed, to rebuild the entire thing.)

--New build system handles this

libraries and flags

There should be a way of specifying, in the pyx/pxd file itself, what libraries to link and/or what compile flags to use, and stuff like if this is C or C++.

Module names should be detected automatically from the file names/directory structure.

Perhaps one shouldn't have to list all .pxd files in setup.py at all--couldn't they be automatically found? (Maybe in all directories that have __init__.py, just like with .py files?)

--New build system handles this