build_ext does not properly track dependencies (and should be deprecated / removed)

[navytux]

Hello up there. In the context of getting gevent miscompiled the question of proper dependency tracking on Cython side was raised. While original cause for that particular gevent miscompilation seems to be #1428, other issues were also found.

This issue is probably related to #1436 and shows that Cython.Distutils.build_ext does not properly track build dependencies.

In 2016 Cython.Distutils.build_ext was changed to use cythonize and the old implementation was moved into Cython.Distutils.old_build_ext for exactly particular reason that old_build_ext was not tracking dependencies properly: cb55c11. A warning corresponding to the move was added:

cython/Cython/Distutils/old_build_ext.py

Lines 37 to 42 in 3de7a4b

	if (not _check_stack('setuptools/extensions.py')
	and not _check_stack('pyximport/pyxbuild.py')
	and not _check_stack('Cython/Distutils/build_ext.py')):
	warnings.warn(
	"Cython.Distutils.old_build_ext does not properly handle dependencies "
	"and is deprecated.")

However right after that Cython.Distutils.build_ext was reverted to use old_build_ext and new implementation became available as new_build_ext (4ecdd3e) with the idea to

give projects more time to move over, and reduces the number of changes in the 0.25 series

As of today (2020 April) Cython.Distutils.build_ext still points to old_build_ext and the warning is somehow not printed even if old_build_ext is imported directly(*):

$ python -c 'import Cython.Distutils.old_build_ext'
# empty output
$ python -c 'from Cython.Build import build_ext'
# empty output

This way many projects still use old_build_ext = from Cython.Build import build_ext to compile their pyx files and miss both proper dependency tracking and the warning.

As a fix I propose to make the deprecation warnings effective and to make Cython.Build.build_ext to refer to new_build_ext which uses cythonize.

The rest of this issue demonstrates that Cython.Build.build_ext dependency handling is broken.

---

setup.py

from setuptools import setup, Extension
from Cython.Distutils import build_ext

setup(
    ext_modules = [
                Extension("a", ["a.pyx"]),
                Extension("b", ["b.pyx"]),
            ],

    cmdclass = {'build_ext': build_ext}
)

a.pyx

# cython: language_level=2
  
from b cimport bfunc

def afunc():
    bfunc()

b.pxd

cdef bfunc()

b.pyx

# cython: language_level=2

cdef bfunc():
    print "bbb"

( first compile - ok )

$ python setup.py build_ext -i
running build_ext
cythoning a.pyx to a.c
cythoning b.pyx to b.c
...

now change b.pxd and b.pyx to add argument to bfunc:

--- a/b.pxd
+++ b/b.pxd
@@ -1 +1 @@
-cdef bfunc()
+cdef bfunc(int x)
--- a/b.pyx
+++ b/b.pyx
@@ -1,4 +1,4 @@
 # cython: language_level=2
 
-cdef bfunc():
-    print "bbb"
+cdef bfunc(int x):
+    print "bbb", x

and recompile - only b is rebuilt - not a:

(neo) (z-dev) (g.env) kirr@deco:~/tmp/trashme/pyx$ python setup.py build_ext -i
running build_ext
skipping 'a.c' Cython extension (up-to-date)      <-- NOTE
cythoning b.pyx to b.c
/home/kirr/src/tools/py/cython/Cython/Compiler/Main.py:344: FutureWarning: Cython directive 'language_level' not set, using '3str' for now (Py3). This has changed from earlier releases! File: /home/kirr/tmp/trashme/pyx/b.pxd
  tree = Parsing.p_module(s, pxd, full_module_name)
building 'b' extension
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-2.7.16=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/python2.7 -c b.c -o build/temp.linux-x86_64-2.7/b.o
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-2.7.16=. -fstack-protector-strong -Wformat -Werror=format-security -Wl,-z,relro -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-2.7.16=. -fstack-protector-strong -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/b.o -o /home/kirr/tmp/trashme/pyx/b.so

which is WRONG because a will use b module via outdated pxd.

( In this particular case it should be giving compilation error at Cython level:

$ touch a.pyx 
$ python setup.py build_ext -i
running build_ext
cythoning a.pyx to a.c

Error compiling Cython file:
------------------------------------------------------------
...
# cython: language_level=2

from b cimport bfunc

def afunc():
    bfunc()
        ^
------------------------------------------------------------

a.pyx:6:9: Call with wrong number of arguments (expected 1, got 0)

)

Cython 3.0a1-79-g3de7a4b8f

Thanks beforehand,
Kirill

/cc @robertwb

P.S.

Cython documentation says to use Cython.Build.new_build_ext which should be using cythonize internally:

Another option is to make Cython a setup dependency of your system and use Cython’s build_ext module which runs cythonize as part of the build process:

setup(
    extensions = [Extension("*", ["*.pyx"])],
    cmdclass={'build_ext': Cython.Build.new_build_ext},
    ...
)

however Cython.Build does not have it:

>>> import Cython.Build
>>> Cython.Build.new_build_ext
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute 'new_build_ext'

because new_build_ext is defined only in Cython.Distutils.build_ext

---

(*) probably due to some logic error in _check_stack calls. if I patch Cython locally with

--- a/Cython/Distutils/old_build_ext.py
+++ b/Cython/Distutils/old_build_ext.py
@@ -34,9 +34,7 @@ def _check_stack(path):
     pass
   return False
 
-if (not _check_stack('setuptools/extensions.py')
-    and not _check_stack('pyximport/pyxbuild.py')
-    and not _check_stack('Cython/Distutils/build_ext.py')):
+if 1:
     warnings.warn(
         "Cython.Distutils.old_build_ext does not properly handle dependencies "
         "and is deprecated.")

the warning is printed:

$ python -c 'import Cython.Distutils.old_build_ext'
/home/kirr/src/tools/py/cython/Cython/Distutils/old_build_ext.py:39: UserWarning: Cython.Distutils.old_build_ext does not properly handle dependencies and is deprecated.
  "Cython.Distutils.old_build_ext does not properly handle dependencies "
$ python -c 'from Cython.Build import build_ext'
/home/kirr/src/tools/py/cython/Cython/Distutils/old_build_ext.py:39: UserWarning: Cython.Distutils.old_build_ext does not properly handle dependencies and is deprecated.
  "Cython.Distutils.old_build_ext does not properly handle dependencies "

[scoder]

Thanks for the report. The thing is, new_build_ext is not compatible with the setup that many users probably configured for old_build_ext, specifically any command options.

What might work, I think, is to copy the current configuration options from old_build_ext to new_build_ext, and pass them over to cythonize() only if they were provided. Basically, integrate a legacy layer into new_build_ext that allows existing projects to build as before. We can issue a warning when we find legacy usages, but we shouldn't break the build for it.

PR very welcome.

[goldenrockefeller]

Here is how the options for the old_build_ext (on the left hand side) roughly map to cythonize parameters (right hand side) used by the new_build_ext:

cython_cplus : language
cython_create_listing : N/A?
cython_line_directives : N/A?
cython_include_dirs : include_path (maybe?)
cython_directives : compiler_directives
cython_c_in_temp : N/A?
cython_gen_pxi : N/A?
cython_gdb : N/A?
no_c_in_traceback : N/A?
cython_compile_time_env : N/A?

Given how incompatible the two build_exts are, is it worth making the new_build_ext backwards compatible with the old_build_ext, or should we just switch to the new_build_ext as a possibly breaking change for Cython 3.0?