PackageHierarchy

How To Create A Hierarchy Of Modules In A Package

Summary

• Set up your source directory with .pyx, .py, and __init__.py files as you normally would.
• Make sure to add "." to the include_dirs list in the Extension constructor call.

This should just work.

Introduction

This page is based on my own experiences writing the DVEdit video editing framework, and discusses some of the practicalities of laying out a hierarchy of extension modules in a Cython project, where such modules are cimporting stuff from other modules.

It initially took me many hours to get it right, by reading docs, and trial and error. Hopefully this page will make it easier for you.

The Problem

I wanted to create the following package hierarchy:

dvedit.core               - cython extension
dvedit.clipview           - pure python
dvedit.filters.inverse    - cython extension, uses cimport'ed defs from dvedit.core
dvedit.filters.flip       - ditto
dvedit.filters.reverse    - ditto

The tricky thing was that the dvedit.filters.* extension modules do a cimport of class definitions from dvedit.core. Initially, when compiling, I couldn't get Cython to find the .pxd file for dvedit.core. Even if I got it all to compile, then I couldn't get all the modules to load at run-time.

The Solution

The area of module hierarchy within packages is one where Cython and Pyrex differ markedly. Pyrex insists that .pyx and .pxd files be named according to their position in the hierarchy, while Cython allows the .pyx files to exist under their 'leaf names' at the appropriate place in a package directory (just like pure-python modules, java source files etc).

So for Pyrex, you'd have all the files sitting flat together in a directory:

dvedit.core.pxd
dvedit.core.pyx
dvedit.filters.inverse.pyx
dvedit.filters.flip.pyx
dvedit.filters.reverse.pyx

Also, you'd have to have a separate directory tree containing the pure-python components:

dvedit/
  __init__.py
  clipview.py
  filters/
    __init__.py

I originally tried that, and built the whole framework under Pyrex. Built and installed fine, and initially seemed to work. However, for some strange reason, the following script would cause a SEGV and much stack corruption:

from dvedit.core import *
from dvedit.filters.inverse import InverseFilter
from dvedit.filters.flip import FlipFilter
from dvedit.filters.reverse import ReverseFilter

And, the crash always happened on the import of the third dvedit.filters.* module. No matter how I simplified dvedit.core.pyx and each of the dvedit.filters.*.pyx files, the problem kept happening. So maybe that was the gods telling me that now is a good time to commit to Cython.

Under Cython, this module hierarchy is laid out exactly as for pure-python files:

dvedit/
  __init__.py
  core.pxd
  core.pyx
  clipview.py
  filters/
    __init__.py
    inverse.pyx
    flip.pyx
    reverse.pyx

Much simpler.

However, I couldn't get it to build under distutils. It took a bit of time sifting through Pyrex source, and sticking in print statements here and there, to figure out the bleedingly obvious solution - adding "." to the include_dirs list in the Extension constructor call. So here's the setup.py file I'm using to build all this. There's a tiny amount of sophistication added - my setup.py file does a recursive scan of the dvedit directory to find all the .pyx files, and creates Extension objects for each of them.

# build script for 'dvedit' - Python libdv wrapper

# change this as needed
libdvIncludeDir = "/usr/include/libdv"

import sys, os
from distutils.core import setup
from distutils.extension import Extension

# we'd better have Cython installed, or it's a no-go
try:
    from Cython.Distutils import build_ext
except:
    print("You don't seem to have Cython installed. Please get a")
    print("copy from www.cython.org and install it")
    sys.exit(1)


# scan the 'dvedit' directory for extension files, converting
# them to extension names in dotted notation
def scandir(dir, files=[]):
    for file in os.listdir(dir):
        path = os.path.join(dir, file)
        if os.path.isfile(path) and path.endswith(".pyx"):
            files.append(path.replace(os.path.sep, ".")[:-4])
        elif os.path.isdir(path):
            scandir(path, files) 
    return files


# generate an Extension object from its dotted name
def makeExtension(extName):
    extPath = extName.replace(".", os.path.sep)+".pyx"
    return Extension(
        extName,
        [extPath],
        include_dirs = [libdvIncludeDir, "."],   # adding the '.' to include_dirs is CRUCIAL!!
        extra_compile_args = ["-O3", "-Wall"],
        extra_link_args = ['-g'],
        libraries = ["dv",],
        )

# get the list of extensions
extNames = scandir("dvedit")

# and build up the set of Extension objects
extensions = [makeExtension(name) for name in extNames]

# finally, we can pass all this to distutils
setup(
  name="dvedit",
  packages=["dvedit", "dvedit.filters"],
  ext_modules=extensions,
  cmdclass = {'build_ext': build_ext},
)

Again, adding "." to the include_dirs array within each Extension constructor call is crucial. Otherwise Cython won't know how to search for the file dvedit/core.pxd when other modules do from dvedit.core cimport ...

Note also the packages list in the setup() call above. We need to state not just the dvedit directory, but all subdirectories within it which contain files needing to be part of the distribution.

Anyway, when we run this setup.py script, we end up with a final package directory:

dvedit
dvedit/__init__.py
dvedit/__init__.pyc
dvedit/core.so
dvedit/clipview.py
dvedit/clipview.pyc
dvedit/filters
dvedit/filters/__init__.py
dvedit/filters/__init__.pyc
dvedit/filters/inverse.so
dvedit/filters/flip.so
dvedit/filters/reverse.so

which simply Just Works.

Conclusion

This page has, using a real example, illustrated how to create a working distribution package containing a hierarchy of Cython and pure-Python modules.

One non-intuitive pitfall was the need to put "." into the include_dirs list in each Extension constructor call, so that the Cython compiler could find the needed .pxd files when various modules perform a cimport. Another pitfall was the need to list all submodule names in the packages list in the final setup() call. But with these details addressed, Cython just gets on with the job of building a perfectly working package tree of interdependent extension modules and pure-python modules.

Discussion

Some attached files: <<AttachList>>