Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Segmentation fault on cythonizing #2199

Closed
wkerzendorf opened this issue Apr 5, 2018 · 16 comments
Closed

Segmentation fault on cythonizing #2199

wkerzendorf opened this issue Apr 5, 2018 · 16 comments
Milestone

Comments

@wkerzendorf
Copy link

We are trying to cythonize our code (github.com/tardis-sn/tardis) within our TRAVIS environment and we get a segmentation fault during the cythonizing (cython 0.27.3 and 0.28) step (see https://travis-ci.org/tardis-sn/tardis/jobs/362581597).

Here's the log:

Compiling tardis/montecarlo/montecarlo.pyx because it changed.
[1/1] Cythonizing tardis/montecarlo/montecarlo.pyx
/home/travis/.travis/job_stages: line 57:  4789 Segmentation fault      (core dumped) CC=$COMPILER python setup.py $SETUP_CMD

I'm also pinging @unoebauer as he was helping to debug this problem

@scoder
Copy link
Contributor

scoder commented Apr 8, 2018

Could you try installing Cython from a PyPI wheel instead of conda?

@scoder
Copy link
Contributor

scoder commented Apr 12, 2018

Do I understand correctly that you cannot reproduce this locally, but only on travis?
I tried compiling that .pyx file myself and (unsurprisingly) couldn't get it to crash. And the file looks rather unsuspicious, definitely not something that could run into resource limits or so.

Ok, travis uses an old Ubuntu version, but apart from that, it's entirely non-obvious to me what might trigger a crash here. I've never seen something like this in years. But I admit that I also don't remember seeing anyone use conda on travis yet.

@wkerzendorf
Copy link
Author

@scoder I've seen many people use conda on TRAVIS - but I have not seen that bug. Not sure where it comes from as we also can not reproduce it locally. I filed a bug on TRAVIS - maybe they can help.

@wkerzendorf
Copy link
Author

wkerzendorf commented May 18, 2018

@scoder - I think we are getting closer. TRAVIS's @BanzaiMan gave me access to the debug mode and I got a core dump:

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f17a0579ea4 in _PyObject_GetDictPtr () from /home/travis/miniconda/envs/tardis/bin/../lib/libpython2.7.so.1.0
#1  0x00007f1797273291 in __Pyx_AddTraceback () from /home/travis/miniconda/envs/tardis/lib/python2.7/site-packages/Cython/Compiler/Code.so
#2  0x00007f179732a726 in __pyx_pw_6Cython_8Compiler_4Code_11GlobalState_61lookup_filename ()
   from /home/travis/miniconda/envs/tardis/lib/python2.7/site-packages/Cython/Compiler/Code.so
#3  0x00007f1797279c45 in __pyx_pw_6Cython_8Compiler_4Code_11CCodeWriter_205lookup_filename ()
   from /home/travis/miniconda/envs/tardis/lib/python2.7/site-packages/Cython/Compiler/Code.so
#4  0x00007f17972ed5a9 in __pyx_pw_6Cython_8Compiler_4Code_11CCodeWriter_195error_goto ()
   from /home/travis/miniconda/envs/tardis/lib/python2.7/site-packages/Cython/Compiler/Code.so
#5  0x00007f179727b8e7 in __pyx_pw_6Cython_8Compiler_4Code_11CCodeWriter_197error_goto_if ()
   from /home/travis/miniconda/envs/tardis/lib/python2.7/site-packages/Cython/Compiler/Code.so
#6  0x00007f17a053d7a3 in PyObject_Call () from /home/travis/miniconda/envs/tardis/bin/../lib/libpython2.7.so.1.0

scoder added a commit that referenced this issue May 21, 2018
@scoder
Copy link
Contributor

scoder commented May 21, 2018

The traceback looks like the problem could at least be worked around by explicitly disabling the feature of including the C code lines in tracebacks at runtime. When building Cython itself, set the C macro -D CYTHON_CLINE_IN_TRACEBACK=0. We might want to set this ourselves in the setup.py.

I found a case in the code that could trigger such a crash, a use before init on very early errors during the module initialisation, and pushed a fix. Would be nice if you could try the latest master build somehow.

@wkerzendorf
Copy link
Author

I definitely can try this - it’s on the master right?

@wkerzendorf
Copy link
Author

@scoder I've used the cython master in this build and it still crashes: https://travis-ci.org/tardis-sn/tardis/jobs/381891389

@wkerzendorf
Copy link
Author

python setup.py build_ext -D CYTHON_CLINE_IN_TRACEBACK=0

produced this error:

[ 6/13] Cythonizing /home/travis/cython/Cython/Compiler/Pythran.py
[ 7/13] Cythonizing /home/travis/cython/Cython/Compiler/Scanning.py
[ 8/13] Cythonizing /home/travis/cython/Cython/Compiler/Visitor.py
[ 9/13] Cythonizing /home/travis/cython/Cython/Plex/Actions.py
[10/13] Cythonizing /home/travis/cython/Cython/Plex/Scanners.py
[11/13] Cythonizing /home/travis/cython/Cython/Runtime/refnanny.pyx
[12/13] Cythonizing /home/travis/cython/Cython/StringIOTree.py
[13/13] Cythonizing /home/travis/cython/Cython/Tempita/_tempita.py
building 'Cython.Plex.Scanners' extension
creating build
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/home
creating build/temp.linux-x86_64-2.7/home/travis
creating build/temp.linux-x86_64-2.7/home/travis/cython
creating build/temp.linux-x86_64-2.7/home/travis/cython/Cython
creating build/temp.linux-x86_64-2.7/home/travis/cython/Cython/Plex
gcc -pthread -B /home/travis/miniconda/envs/tardis/compiler_compat -Wl,--sysroot=/ -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DCYTHON_CLINE_IN_TRACEBACK=0=1 -I/home/travis/miniconda/envs/tardis/include/python2.7 -c /home/travis/cython/Cython/Plex/Scanners.c -o build/temp.linux-x86_64-2.7/home/travis/cython/Cython/Plex/Scanners.o
/home/travis/cython/Cython/Plex/Scanners.c: In function__Pyx_AddTraceback’:
<command-line>:0:28: error: lvalue required as left operand of assignment
/home/travis/cython/Cython/Plex/Scanners.c:1465:53: note: in expansion of macroCYTHON_CLINE_IN_TRACEBACK#define __Pyx_CLineForTraceback(tstate, c_line)  (((CYTHON_CLINE_IN_TRACEBACK)) ? c_line : 0)
                                                     ^
/home/travis/cython/Cython/Plex/Scanners.c:11195:18: note: in expansion of macro__Pyx_CLineForTracebackc_line = __Pyx_CLineForTraceback(tstate, c_line);
                  ^
error: command 'gcc' failed with exit status 1

@scoder
Copy link
Contributor

scoder commented May 22, 2018

I meant something like CFLAGS="$CFLAGS -D CYTHON_CLINE_IN_TRACEBACK=0".

@wkerzendorf
Copy link
Author

wkerzendorf commented May 22, 2018

@scoder that worked - it doesn't crash. What's the best way to get this into the travis?

@wkerzendorf
Copy link
Author

@scoder we integrated this into travis in the following way: https://github.com/tardis-sn/tardis/blob/master/.travis.yml (using the compile flags). At what point can we use a standard release version of cython again?

@scoder
Copy link
Contributor

scoder commented Jul 18, 2018

Hmm, looks like the fix didn't make it into the changelog. It's included in 0.28.3, though.

@wkerzendorf
Copy link
Author

@wkerzendorf
Copy link
Author

wkerzendorf commented Jul 18, 2018

it's the OSX version - any idea why that would causing trouble?

@wkerzendorf
Copy link
Author

@scoder this looks like the same bug. It occurs on OSX with clang using cython 0.28.4:

* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
  * frame #0: 0x00000001000e4608 libpython2.7.dylib`_PyObject_GetDictPtr + 8
    frame #1: 0x0000000103d1e27b Code.so`__Pyx_AddTraceback + 107
    frame #2: 0x0000000103d8b234 Code.so`__pyx_pw_6Cython_8Compiler_4Code_11GlobalState_61lookup_filename + 1924
    frame #3: 0x0000000103d3b80b Code.so`__pyx_pw_6Cython_8Compiler_4Code_11CCodeWriter_205lookup_filename + 379
    frame #4: 0x0000000103d3df31 Code.so`__pyx_pw_6Cython_8Compiler_4Code_11CCodeWriter_195error_goto + 1137
    frame #5: 0x0000000103d3d387 Code.so`__pyx_pw_6Cython_8Compiler_4Code_11CCodeWriter_197error_goto_if + 759
    frame #6: 0x000000010009dc55 libpython2.7.dylib`PyObject_Call + 101
    frame #7: 0x000000010014a168 libpython2.7.dylib`PyEval_EvalFrameEx + 22088
    frame #8: 0x00000001001448c4 libpython2.7.dylib`PyEval_EvalCodeEx + 2164
    frame #9: 0x000000010014ec0d libpython2.7.dylib`fast_function + 109

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants