Skip to content
This repository

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

SCons dependency tracking support #11

Open
wants to merge 1 commit into from

2 participants

Holger Rapp Robert Bradshaw
Holger Rapp

Hey Folks,

I've written a dependency tracker for cython. The tracker is pretty much stand alone in an extra file (including tests) and can also be easily used in other tools.

I enhanced the scons support that was already there by hooking my tracker into it. I am using this to build my scientific code. I also added a getting started README for new scons users that should get them up to speed quickly.

What do you think?

Btw, glad that you are on github now. Git makes more sense to me than hg does :).

Holger Rapp

Robert Bradshaw
Owner

Cool. My only thought is that it would make more sense to leverage the dependency checker that's already part of Cython (https://github.com/cython/cython/blob/master/Cython/Build/Dependencies.py), rather than having two implementations. The latter correctly handles cases like

my_docstring = """
cimport this module
"""

or pxi files that contain cimports (which are resolved relative to the including file, rather than relative to the included file). You could then also leverage inline build options "distutils: option=value" for scons.

Holger Rapp

robert,

I wasn't aware of this implementation. Sadly it is not documented (afaik) and it is not trivial for me where the entry point is. Could you help me out which function returns the dependant files?

Robert Bradshaw
Owner

Look in Cython/Build/Dependencies.py, in particular create_dependency_tree(). The context is primarily to set up the correct include paths, see the cythonize() function for an entry point. It could perhaps be cleaned up a bit to better fit into your model, but would be code well worth leveraging.

Holger Rapp

I invested an hour now in this. I ran into bugs with the dependency tree, for my source files dt.cimported_files(fn) recurses infinitively. I also realized that the dependency code also just uses regular expression with some preprocessing that feels unsafe, so I see no big benefits in using this and I also do not see where it is superior to my implementation. I also think my implementation is easier to work with because it comes with tests. So, I will not invest more time in turning my scons dependency tracker into using the Cython Dependency tracker. Whoever is working on this might consider taking a peek at my tracker.

I would also need a way to tell the Context that we are working from another cwd. making a chdir and back is the only way I got this working, but this is error prone.

So feel free to merge my enhancements to the scons tool or discard this pull request. I will not work on it any further for the moment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Showing 1 unique commit by 1 author.

Mar 28, 2011
Holger Rapp Enhanced Scons support by adding dependency tracking e53de3c
This page is out of date. Refresh to see the latest.
38 Tools/site_scons/site_tools/README_cython.txt
... ... @@ -0,0 +1,38 @@
  1 +Howto build cython code with Scons
  2 +==================================
  3 +
  4 +Quickstart
  5 +----------
  6 +
  7 +My own entry points was [1]. Copy this directory into your project under::
  8 +
  9 + project_root/site_scons/site_tools/
  10 +
  11 +Now add to your top level SConscript::
  12 +
  13 + env = Environment(PYEXT_USE_DISTUTILS=True)
  14 +
  15 + env.Tool("pyext")
  16 + env.Tool("cython")
  17 +
  18 +You can now build Cython Extensions using the PythonExtension builder::
  19 +
  20 + env.PythonExtension("blah", ["blah.pyx"])
  21 +
  22 +The builder is aware of cython dependencies residing in the search paths
  23 +given by CYTHONPATH. Dependency tracking is then automatically done by scons.
  24 +
  25 +[1] http://www.mail-archive.com/cython-dev@codespeak.net/msg09540.html
  26 +
  27 +Configuration
  28 +-------------
  29 +
  30 +The builders define some configuration variables which you can set like so::
  31 +
  32 + env.Append(CYTHONPATH = ["thisdirectory"]) # add a directory to cythons search path
  33 +
  34 +Have a look at cython.py and pyext.py for finding them.
  35 +
  36 +
  37 +
  38 +-- vim:ft=rst
97 Tools/site_scons/site_tools/_cython_dependencies.py
... ... @@ -0,0 +1,97 @@
  1 +#!/usr/bin/env python
  2 +# encoding: utf-8
  3 +
  4 +import re
  5 +import os
  6 +import operator
  7 +
  8 +__all__ = ["CythonDependencyScanner"]
  9 +
  10 +flatten = lambda a: reduce(operator.add, a, [])
  11 +class CythonDependencyScanner(object):
  12 + # The regular expression were originally from the sage setup.py
  13 + _DEP_REGS_PXD = [
  14 + re.compile(r'^ *(?:cimport +([\w\. ,]+))', re.M),
  15 + re.compile(r'^ *(?:from +([\w.]+) +cimport)', re.M),
  16 + ]
  17 + _DEP_REG_DIRECT = \
  18 + re.compile(r'^ *(?:include *[\'"]([^\'"]+)[\'"])', re.M)
  19 + _DEP_REG_CHEADER = \
  20 + re.compile(r'^ *(?:cdef[ ]*extern[ ]*from *[\'"]([^\'"]+)[\'"])', re.M)
  21 +
  22 + def __init__(self, include_paths):
  23 + self.deps = set()
  24 + self.include_paths = include_paths
  25 +
  26 + def _find_deps(self, s):
  27 + self._find_deps_pxd(s)
  28 + self._find_deps_cheader(s)
  29 + self._find_deps_direct(s)
  30 +
  31 + @staticmethod
  32 + def _normalize_module_name(s):
  33 + # Remove as blah at the end
  34 + s = s.split(" as ")[0].strip()
  35 +
  36 + # Replace all dots except a path seperator
  37 + s = s.replace('.', os.path.sep)
  38 +
  39 + return s
  40 +
  41 + def _find_deps_pxd(self, s):
  42 + temp = flatten([m.findall(s) for m in self._DEP_REGS_PXD])
  43 + all_matches = flatten([ [ s.strip() for s in m.split(',') ] for m in temp])
  44 +
  45 + for dep in all_matches:
  46 + dep = self._normalize_module_name(dep)
  47 + dep += '.pxd'
  48 + if dep not in self.deps:
  49 + # Recurse, if file exists. If not, the file might be global
  50 + # (which we currently do not track) or the file
  51 + # might not exist, which is not our problem, but cythons
  52 + for path in self.include_paths + ['']:
  53 + filename = os.path.relpath(os.path.join(path, dep))
  54 + if os.path.exists(filename):
  55 + self._find_deps(open(filename,"r").read())
  56 + self.deps.add(filename)
  57 +
  58 + def _find_deps_direct(self, s):
  59 + all_matches = self._DEP_REG_DIRECT.findall(s)
  60 +
  61 + for dep in all_matches:
  62 + if dep not in self.deps:
  63 + # Recurse, if file exists. If not, the file might be global
  64 + # (which we currently do not track) or the file
  65 + # might not exist, which is not our problem, but cythons
  66 + for path in self.include_paths + ['']:
  67 + filename = os.path.relpath(os.path.join(path, dep))
  68 + if os.path.exists(filename):
  69 + self._find_deps(open(filename,"r").read())
  70 + self.deps.add(filename)
  71 +
  72 + def _find_deps_cheader(self, s):
  73 + all_matches = self._DEP_REG_CHEADER.findall(s)
  74 +
  75 + for dep in all_matches:
  76 + if dep not in self.deps:
  77 + # Recurse, if file exists. If not, the file might be global
  78 + # (which we currently do not track) or the file
  79 + # might not exist, which is not our problem, but cythons
  80 + # TODO: we should track the headers included by this
  81 + # header. But currently we don't
  82 + self.deps.add(dep)
  83 +
  84 + def __call__(self, source):
  85 + self.deps = set()
  86 +
  87 + # We might also depend on our definition file if it exists
  88 + pxd = os.path.splitext(source)[0] + '.pxd'
  89 + if os.path.exists(pxd):
  90 + self.deps.add(pxd)
  91 + self._find_deps(open(pxd, "r").read())
  92 +
  93 + self._find_deps(open(source,"r").read())
  94 + self.deps = [ d for d in self.deps if os.path.exists(d) ]
  95 +
  96 + return self.deps
  97 +
35 Tools/site_scons/site_tools/cython.py
@@ -14,16 +14,24 @@
14 14 AUTHORS:
15 15 - David Cournapeau
16 16 - Dag Sverre Seljebotn
  17 + - Holger Rapp (HolgerRapp@gmx.net)
17 18
18 19 """
  20 +import os.path
  21 +
19 22 import SCons
20   -from SCons.Builder import Builder
21 23 from SCons.Action import Action
  24 +from SCons.Builder import Builder
  25 +from SCons.Scanner import Scanner, FindPathDirs
  26 +from SCons.Tool import SourceFileScanner
  27 +
  28 +from _cython_dependencies import CythonDependencyScanner
22 29
23   -#def cython_action(target, source, env):
24   -# print target, source, env
25   -# from Cython.Compiler.Main import compile as cython_compile
26   -# res = cython_compile(str(source[0]))
  30 +def cython_dependency_scanner(node, env, path, arg):
  31 + cdir = os.path.dirname(str(node))
  32 + scanner = CythonDependencyScanner([cdir] + map(str,path))
  33 +
  34 + return [os.path.relpath(f, cdir) for f in scanner(str(node))]
27 35
28 36 cythonAction = Action("$CYTHONCOM")
29 37
@@ -35,6 +43,7 @@ def create_builder(env):
35 43 action = cythonAction,
36 44 emitter = {},
37 45 suffix = cython_suffix_emitter,
  46 + source_scanner = SourceFileScanner,
38 47 single_source = 1)
39 48 env['BUILDERS']['Cython'] = cython
40 49
@@ -45,7 +54,11 @@ def cython_suffix_emitter(env, source):
45 54
46 55 def generate(env):
47 56 env["CYTHON"] = "cython"
48   - env["CYTHONCOM"] = "$CYTHON $CYTHONFLAGS -o $TARGET $SOURCE"
  57 + env["CYTHONPATH"] = []
  58 + env['CYTHONINCPREFIX'] = '-I'
  59 + env['CYTHONINCSUFFIX'] = ''
  60 + env['_CYTHONINCFLAGS'] = '$( ${_concat(CYTHONINCPREFIX, CYTHONPATH, CYTHONINCSUFFIX, __env__, RDirs, TARGET, SOURCE)} $)'
  61 + env["CYTHONCOM"] = "$CYTHON ${_CYTHONINCFLAGS} $CYTHONFLAGS -o $TARGET $SOURCE"
49 62 env["CYTHONCFILESUFFIX"] = ".c"
50 63
51 64 c_file, cxx_file = SCons.Tool.createCFileBuilders(env)
@@ -58,6 +71,16 @@ def generate(env):
58 71
59 72 create_builder(env)
60 73
  74 + __scanner = Scanner(name = 'cython',
  75 + function = cython_dependency_scanner,
  76 + argument = None,
  77 + skeys = ['.pyx', '.pxd'],
  78 + path_function = FindPathDirs("CYTHONPATH"),
  79 + recursive = False) # CythonDependencyScanner recurses automatically
  80 +
  81 + env.Append(SCANNERS = __scanner)
  82 +
  83 +
61 84 def exists(env):
62 85 try:
63 86 # import Cython
86 Tools/site_scons/site_tools/tests/test_cython.py
... ... @@ -0,0 +1,86 @@
  1 +#!/usr/bin/env python
  2 +# encoding: utf-8
  3 +
  4 +from nose.tools import *
  5 +
  6 +import sys, os
  7 +sys.path.append(os.path.join(os.path.dirname(__file__), os.path.pardir))
  8 +
  9 +from _cython_dependencies import CythonDependencyScanner
  10 +
  11 +class _CythonTest(object):
  12 + def setUp(self):
  13 + self.c = CythonDependencyScanner([])
  14 +
  15 +
  16 +class Test_CythonDependencyTracking(_CythonTest): # {{{
  17 + def test_simple_from(self):
  18 + self.c._find_deps("""from blub cimport hallo""")
  19 +
  20 + ok_("blub.pxd" in self.c.deps)
  21 + eq_(len(self.c.deps), 1)
  22 + def test_simple_from_submodul(self):
  23 + self.c._find_deps("""from geometry.objects cimport hi""")
  24 +
  25 + ok_("geometry/objects.pxd" in self.c.deps)
  26 + eq_(len(self.c.deps), 1)
  27 +
  28 + def test_simple_cimport(self):
  29 + self.c._find_deps("""cimport hallo""")
  30 +
  31 + ok_("hallo.pxd" in self.c.deps)
  32 + eq_(len(self.c.deps), 1)
  33 +
  34 + def test_simple_cimport_from_submodule(self):
  35 + self.c._find_deps("""cimport hallo.welt""")
  36 +
  37 + ok_("hallo/welt.pxd" in self.c.deps)
  38 + eq_(len(self.c.deps), 1)
  39 +
  40 + def test_multiple_cimport(self):
  41 + self.c._find_deps("""cimport hallo, sunshine""")
  42 +
  43 + ok_("hallo.pxd" in self.c.deps)
  44 + ok_("sunshine.pxd" in self.c.deps)
  45 + eq_(len(self.c.deps), 2)
  46 + def test_cimport_with_as(self):
  47 + self.c._find_deps("""cimport hallo as h""")
  48 +
  49 + ok_("hallo.pxd" in self.c.deps)
  50 + eq_(len(self.c.deps), 1)
  51 +
  52 +
  53 + def test_simple_include(self):
  54 + self.c._find_deps('include "hello.pxi"')
  55 +
  56 + ok_("hello.pxi" in self.c.deps)
  57 + eq_(len(self.c.deps), 1)
  58 +
  59 + def test_simple_cheader(self):
  60 + self.c._find_deps('cdef extern from "hello.h"')
  61 +
  62 + ok_("hello.h" in self.c.deps)
  63 + eq_(len(self.c.deps), 1)
  64 +
  65 + def test_realworld_example(self):
  66 + self.c._find_deps('''
  67 +cdef extern from "math.h":
  68 + double sqrt(double)
  69 +
  70 +from geom cimport Triangle, Rect
  71 +cimport Integrator, House
  72 +
  73 +def import_function():
  74 + pass
  75 +''')
  76 +
  77 + ok_("math.h" in self.c.deps)
  78 + ok_("geom.pxd" in self.c.deps)
  79 +
  80 + ok_("Integrator.pxd" in self.c.deps)
  81 + ok_("House.pxd" in self.c.deps)
  82 +
  83 + eq_(len(self.c.deps), 4)
  84 +
  85 +
  86 +

Tip: You can add notes to lines in a file. Hover to the left of a line to make a note

Something went wrong with that request. Please try again.