cythonize doesn’t recompile when only the pxd has changed in pure python mode

[robertwb]

When a .py file is augmented by a .pxd one, cythonize won’t recompile the module when only the pxd has changed, or one of its dependencies, which often leads to broken code.

This doesn’t happen when the base module is a .pyx one.

Migrated from http://trac.cython.org/ticket/874

[navytux]

This is still true as of today (cython 3.0a1-79-g3de7a4b8f):

$ ls
a.pxd  a.py  setup.py

$ cat setup.py

from setuptools import setup
from Cython.Build import cythonize

setup(
    ext_modules = cythonize(["a.py"], language_level=3)
)

$ cat a.py

def afunc(x):
    return x+2

$ cat a.pxd

cdef int afunc(int x)

( first time compile -> ok )

$ python setup.py build_ext -i
Compiling a.py because it changed.
[1/1] Cythonizing a.py
running build_ext
building 'a' extension
creating build
creating build/temp.linux-x86_64-2.7
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-2.7.16=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/python2.7 -c a.c -o build/temp.linux-x86_64-2.7/a.o
creating build/lib.linux-x86_64-2.7
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-2.7.16=. -fstack-protector-strong -Wformat -Werror=format-security -Wl,-z,relro -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-2.7.16=. -fstack-protector-strong -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/a.o -o build/lib.linux-x86_64-2.7/a.so
copying build/lib.linux-x86_64-2.7/a.so -> 

( second time compile -> nothing is done (ok) )

$ python setup.py build_ext -i
running build_ext
copying build/lib.linux-x86_64-2.7/a.so -> 

( change a.py + compile -> a.so recompiled (ok) )

$ touch a.py 
$ python setup.py build_ext -i
Compiling a.py because it changed.
[1/1] Cythonizing a.py
running build_ext
building 'a' extension
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-2.7.16=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/python2.7 -c a.c -o build/temp.linux-x86_64-2.7/a.o
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-2.7.16=. -fstack-protector-strong -Wformat -Werror=format-security -Wl,-z,relro -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-2.7.16=. -fstack-protector-strong -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/a.o -o build/lib.linux-x86_64-2.7/a.so
copying build/lib.linux-x86_64-2.7/a.so -> 

( change a.pxd+ compile -> a.so is not recompiled (WRONG) )

$ touch a.pxd 
$ python setup.py build_ext -i
running build_ext
copying build/lib.linux-x86_64-2.7/a.so -> 

( a.so is not recompiled even if we change content of a.pxd (WRONG) )

$ echo ZZZ >a.pxd 
$ python setup.py build_ext -i
running build_ext
copying build/lib.linux-x86_64-2.7/a.so -> 

( it recompiles and gives the error only if a.py is changed )

$ touch a.py 
$ python setup.py build_ext -i
Compiling a.py because it changed.
[1/1] Cythonizing a.py

Error compiling Cython file:
------------------------------------------------------------
...
ZZZ
^
------------------------------------------------------------

a.pxd:1:0: undeclared name not builtin: ZZZ
Traceback (most recent call last):
  File "setup.py", line 5, in <module>
    ext_modules = cythonize(["a.py"], language_level=3)
  File "/home/kirr/src/tools/py/cython/Cython/Build/Dependencies.py", line 1105, in cythonize
    cythonize_one(*args)
  File "/home/kirr/src/tools/py/cython/Cython/Build/Dependencies.py", line 1263, in cythonize_one
    raise CompileError(None, pyx_file)
Cython.Compiler.Errors.CompileError: a.py

---

This issue was hit for real with gevent where it was not evident why a code was crashing and only later it was discovered be a miscompilation issue: gevent/gevent#1568 (comment) .

[scoder]

PR welcome.

[0dminnimda]

Noticed this same behavior in 3.0a7 release.
A small note, also if we have a test.py file that we already compiled earlier and we add test.pxd, pxd is also not detected, compilation can only be started by changes in test.py, then test.pxd will be taken into account by the compiler.

I'm not sure if this remark is enough for a separate issue, so I'll post it here.

[scoder]

A small note, also if we have a test.py file that we already compiled earlier and we add test.pxd, pxd is also not detected, compilation can only be started by changes in test.py, then test.pxd will be taken into account by the compiler.

I think this case is covered by the .pxd file being newer than the .c file. Worth a test, obviously.