Segmentation fault on cythonizing

[wkerzendorf]

We are trying to cythonize our code (github.com/tardis-sn/tardis) within our TRAVIS environment and we get a segmentation fault during the cythonizing (cython 0.27.3 and 0.28) step (see https://travis-ci.org/tardis-sn/tardis/jobs/362581597).

Here's the log:

Compiling tardis/montecarlo/montecarlo.pyx because it changed.
[1/1] Cythonizing tardis/montecarlo/montecarlo.pyx
/home/travis/.travis/job_stages: line 57:  4789 Segmentation fault      (core dumped) CC=$COMPILER python setup.py $SETUP_CMD

I'm also pinging @unoebauer as he was helping to debug this problem

[scoder]

Could you try installing Cython from a PyPI wheel instead of conda?

[wkerzendorf]

@scoder we tried - again segfault: https://travis-ci.org/tardis-sn/tardis/jobs/365640859 -
it downloads this wheel:
Downloading https://files.pythonhosted.org/packages/f7/a1/0ecf3ace2e97eaf63f93931be38361763fff0118aca6b70119dde9867e14/Cython-0.28-cp27-cp27mu-manylinux1_x86_64.whl

[scoder]

Do I understand correctly that you cannot reproduce this locally, but only on travis?
I tried compiling that .pyx file myself and (unsurprisingly) couldn't get it to crash. And the file looks rather unsuspicious, definitely not something that could run into resource limits or so.

Ok, travis uses an old Ubuntu version, but apart from that, it's entirely non-obvious to me what might trigger a crash here. I've never seen something like this in years. But I admit that I also don't remember seeing anyone use conda on travis yet.

[wkerzendorf]

@scoder I've seen many people use conda on TRAVIS - but I have not seen that bug. Not sure where it comes from as we also can not reproduce it locally. I filed a bug on TRAVIS - maybe they can help.

[wkerzendorf]

@scoder - I think we are getting closer. TRAVIS's @BanzaiMan gave me access to the debug mode and I got a core dump:

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f17a0579ea4 in _PyObject_GetDictPtr () from /home/travis/miniconda/envs/tardis/bin/../lib/libpython2.7.so.1.0
#1  0x00007f1797273291 in __Pyx_AddTraceback () from /home/travis/miniconda/envs/tardis/lib/python2.7/site-packages/Cython/Compiler/Code.so
#2  0x00007f179732a726 in __pyx_pw_6Cython_8Compiler_4Code_11GlobalState_61lookup_filename ()
   from /home/travis/miniconda/envs/tardis/lib/python2.7/site-packages/Cython/Compiler/Code.so
#3  0x00007f1797279c45 in __pyx_pw_6Cython_8Compiler_4Code_11CCodeWriter_205lookup_filename ()
   from /home/travis/miniconda/envs/tardis/lib/python2.7/site-packages/Cython/Compiler/Code.so
#4  0x00007f17972ed5a9 in __pyx_pw_6Cython_8Compiler_4Code_11CCodeWriter_195error_goto ()
   from /home/travis/miniconda/envs/tardis/lib/python2.7/site-packages/Cython/Compiler/Code.so
#5  0x00007f179727b8e7 in __pyx_pw_6Cython_8Compiler_4Code_11CCodeWriter_197error_goto_if ()
   from /home/travis/miniconda/envs/tardis/lib/python2.7/site-packages/Cython/Compiler/Code.so
#6  0x00007f17a053d7a3 in PyObject_Call () from /home/travis/miniconda/envs/tardis/bin/../lib/libpython2.7.so.1.0

[scoder]

The traceback looks like the problem could at least be worked around by explicitly disabling the feature of including the C code lines in tracebacks at runtime. When building Cython itself, set the C macro -D CYTHON_CLINE_IN_TRACEBACK=0. We might want to set this ourselves in the setup.py.

I found a case in the code that could trigger such a crash, a use before init on very early errors during the module initialisation, and pushed a fix. Would be nice if you could try the latest master build somehow.

[wkerzendorf]

I definitely can try this - it’s on the master right?

[wkerzendorf]

@scoder I've used the cython master in this build and it still crashes: https://travis-ci.org/tardis-sn/tardis/jobs/381891389

[wkerzendorf]

python setup.py build_ext -D CYTHON_CLINE_IN_TRACEBACK=0

produced this error:

[ 6/13] Cythonizing /home/travis/cython/Cython/Compiler/Pythran.py
[ 7/13] Cythonizing /home/travis/cython/Cython/Compiler/Scanning.py
[ 8/13] Cythonizing /home/travis/cython/Cython/Compiler/Visitor.py
[ 9/13] Cythonizing /home/travis/cython/Cython/Plex/Actions.py
[10/13] Cythonizing /home/travis/cython/Cython/Plex/Scanners.py
[11/13] Cythonizing /home/travis/cython/Cython/Runtime/refnanny.pyx
[12/13] Cythonizing /home/travis/cython/Cython/StringIOTree.py
[13/13] Cythonizing /home/travis/cython/Cython/Tempita/_tempita.py
building 'Cython.Plex.Scanners' extension
creating build
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/home
creating build/temp.linux-x86_64-2.7/home/travis
creating build/temp.linux-x86_64-2.7/home/travis/cython
creating build/temp.linux-x86_64-2.7/home/travis/cython/Cython
creating build/temp.linux-x86_64-2.7/home/travis/cython/Cython/Plex
gcc -pthread -B /home/travis/miniconda/envs/tardis/compiler_compat -Wl,--sysroot=/ -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DCYTHON_CLINE_IN_TRACEBACK=0=1 -I/home/travis/miniconda/envs/tardis/include/python2.7 -c /home/travis/cython/Cython/Plex/Scanners.c -o build/temp.linux-x86_64-2.7/home/travis/cython/Cython/Plex/Scanners.o
/home/travis/cython/Cython/Plex/Scanners.c: In function ‘__Pyx_AddTraceback’:
<command-line>:0:28: error: lvalue required as left operand of assignment
/home/travis/cython/Cython/Plex/Scanners.c:1465:53: note: in expansion of macro ‘CYTHON_CLINE_IN_TRACEBACK’
 #define __Pyx_CLineForTraceback(tstate, c_line)  (((CYTHON_CLINE_IN_TRACEBACK)) ? c_line : 0)
                                                     ^
/home/travis/cython/Cython/Plex/Scanners.c:11195:18: note: in expansion of macro ‘__Pyx_CLineForTraceback’
         c_line = __Pyx_CLineForTraceback(tstate, c_line);
                  ^
error: command 'gcc' failed with exit status 1

[scoder]

I meant something like CFLAGS="$CFLAGS -D CYTHON_CLINE_IN_TRACEBACK=0".

[wkerzendorf]

@scoder that worked - it doesn't crash. What's the best way to get this into the travis?

[wkerzendorf]

@scoder we integrated this into travis in the following way: https://github.com/tardis-sn/tardis/blob/master/.travis.yml (using the compile flags). At what point can we use a standard release version of cython again?

[scoder]

Hmm, looks like the fix didn't make it into the changelog. It's included in 0.28.3, though.

[wkerzendorf]

@scoder - it still crashes https://travis-ci.org/tardis-sn/tardis/jobs/405443747

[wkerzendorf]

it's the OSX version - any idea why that would causing trouble?

[wkerzendorf]

@scoder this looks like the same bug. It occurs on OSX with clang using cython 0.28.4:

* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
  * frame #0: 0x00000001000e4608 libpython2.7.dylib`_PyObject_GetDictPtr + 8
    frame #1: 0x0000000103d1e27b Code.so`__Pyx_AddTraceback + 107
    frame #2: 0x0000000103d8b234 Code.so`__pyx_pw_6Cython_8Compiler_4Code_11GlobalState_61lookup_filename + 1924
    frame #3: 0x0000000103d3b80b Code.so`__pyx_pw_6Cython_8Compiler_4Code_11CCodeWriter_205lookup_filename + 379
    frame #4: 0x0000000103d3df31 Code.so`__pyx_pw_6Cython_8Compiler_4Code_11CCodeWriter_195error_goto + 1137
    frame #5: 0x0000000103d3d387 Code.so`__pyx_pw_6Cython_8Compiler_4Code_11CCodeWriter_197error_goto_if + 759
    frame #6: 0x000000010009dc55 libpython2.7.dylib`PyObject_Call + 101
    frame #7: 0x000000010014a168 libpython2.7.dylib`PyEval_EvalFrameEx + 22088
    frame #8: 0x00000001001448c4 libpython2.7.dylib`PyEval_EvalCodeEx + 2164
    frame #9: 0x000000010014ec0d libpython2.7.dylib`fast_function + 109