Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cythonize doesn’t recompile when only the pxd has changed in pure python mode #1428

Closed
robertwb opened this issue Nov 27, 2015 · 5 comments · Fixed by #4063
Closed

cythonize doesn’t recompile when only the pxd has changed in pure python mode #1428

robertwb opened this issue Nov 27, 2015 · 5 comments · Fixed by #4063

Comments

@robertwb
Copy link
Contributor

When a .py file is augmented by a .pxd one, cythonize won’t recompile the module when only the pxd has changed, or one of its dependencies, which often leads to broken code.

This doesn’t happen when the base module is a .pyx one.

Migrated from http://trac.cython.org/ticket/874

@navytux
Copy link
Contributor

navytux commented Apr 22, 2020

This is still true as of today (cython 3.0a1-79-g3de7a4b8f):

$ ls
a.pxd  a.py  setup.py

$ cat setup.py

from setuptools import setup
from Cython.Build import cythonize

setup(
    ext_modules = cythonize(["a.py"], language_level=3)
)

$ cat a.py

def afunc(x):
    return x+2

$ cat a.pxd

cdef int afunc(int x)

( first time compile -> ok )

$ python setup.py build_ext -i
Compiling a.py because it changed.
[1/1] Cythonizing a.py
running build_ext
building 'a' extension
creating build
creating build/temp.linux-x86_64-2.7
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-2.7.16=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/python2.7 -c a.c -o build/temp.linux-x86_64-2.7/a.o
creating build/lib.linux-x86_64-2.7
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-2.7.16=. -fstack-protector-strong -Wformat -Werror=format-security -Wl,-z,relro -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-2.7.16=. -fstack-protector-strong -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/a.o -o build/lib.linux-x86_64-2.7/a.so
copying build/lib.linux-x86_64-2.7/a.so -> 

( second time compile -> nothing is done (ok) )

$ python setup.py build_ext -i
running build_ext
copying build/lib.linux-x86_64-2.7/a.so -> 

( change a.py + compile -> a.so recompiled (ok) )

$ touch a.py 
$ python setup.py build_ext -i
Compiling a.py because it changed.
[1/1] Cythonizing a.py
running build_ext
building 'a' extension
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-2.7.16=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/python2.7 -c a.c -o build/temp.linux-x86_64-2.7/a.o
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-2.7.16=. -fstack-protector-strong -Wformat -Werror=format-security -Wl,-z,relro -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-2.7.16=. -fstack-protector-strong -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/a.o -o build/lib.linux-x86_64-2.7/a.so
copying build/lib.linux-x86_64-2.7/a.so -> 

( change a.pxd+ compile -> a.so is not recompiled (WRONG) )

$ touch a.pxd 
$ python setup.py build_ext -i
running build_ext
copying build/lib.linux-x86_64-2.7/a.so -> 

( a.so is not recompiled even if we change content of a.pxd (WRONG) )

$ echo ZZZ >a.pxd 
$ python setup.py build_ext -i
running build_ext
copying build/lib.linux-x86_64-2.7/a.so -> 

( it recompiles and gives the error only if a.py is changed )

$ touch a.py 
$ python setup.py build_ext -i
Compiling a.py because it changed.
[1/1] Cythonizing a.py

Error compiling Cython file:
------------------------------------------------------------
...
ZZZ
^
------------------------------------------------------------

a.pxd:1:0: undeclared name not builtin: ZZZ
Traceback (most recent call last):
  File "setup.py", line 5, in <module>
    ext_modules = cythonize(["a.py"], language_level=3)
  File "/home/kirr/src/tools/py/cython/Cython/Build/Dependencies.py", line 1105, in cythonize
    cythonize_one(*args)
  File "/home/kirr/src/tools/py/cython/Cython/Build/Dependencies.py", line 1263, in cythonize_one
    raise CompileError(None, pyx_file)
Cython.Compiler.Errors.CompileError: a.py

This issue was hit for real with gevent where it was not evident why a code was crashing and only later it was discovered be a miscompilation issue: gevent/gevent#1568 (comment) .

@scoder
Copy link
Contributor

scoder commented Apr 22, 2020

PR welcome.

@0dminnimda
Copy link
Contributor

Noticed this same behavior in 3.0a7 release.
A small note, also if we have a test.py file that we already compiled earlier and we add test.pxd, pxd is also not detected, compilation can only be started by changes in test.py, then test.pxd will be taken into account by the compiler.

I'm not sure if this remark is enough for a separate issue, so I'll post it here.

@scoder
Copy link
Contributor

scoder commented Jun 30, 2021

A small note, also if we have a test.py file that we already compiled earlier and we add test.pxd, pxd is also not detected, compilation can only be started by changes in test.py, then test.pxd will be taken into account by the compiler.

I think this case is covered by the .pxd file being newer than the .c file. Worth a test, obviously.

@0dminnimda
Copy link
Contributor

0dminnimda commented Jun 30, 2021

A small note, also if we have a test.py file that we already compiled earlier and we add test.pxd, pxd is also not detected, compilation can only be started by changes in test.py, then test.pxd will be taken into account by the compiler.

I think this case is covered by the .pxd file being newer than the .c file. Worth a test, obviously.

This is tested here (recythonize_on_pxd_change) when pxd_exists_for_first_check = False

0dminnimda added a commit to 0dminnimda/cython that referenced this issue Jul 2, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants