free() error with python 2.7

[bnavigator]

Currently the packaging for OpenSUSE fails because somehow free() is called with an invalid pointer when the python 2.7 interpreter terminates.

[  370s] Running unit tests for slycot
[  370s] NumPy version 1.16.5
[  370s] NumPy relaxed strides checking option: True
[  370s] NumPy is installed in /usr/lib64/python2.7/site-packages/numpy
[  370s] Python version 2.7.16 (default, Mar 04 2019, 07:13:50) [GCC]
[  370s] nose version 1.3.7
[  370s] free(): invalid pointer
[  370s] /var/tmp/rpm-tmp.0U6LpC: line 39:  6542 Aborted                 python2 runtests.py --coverage --no-build
[  370s] error: Bad exit status from /var/tmp/rpm-tmp.0U6LpC (%check)
[  370s] 

Minimal example where this occurs is just

abuild@buildmachine:~> python2
Python 2.7.16 (default, Mar 04 2019, 07:13:50) [GCC] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import slycot
>>> exit()
free(): invalid pointer
Aborted (core dumped)

Python3 is fine.

[bnavigator]

This is not restricted to OpenSUSE. I compiled the package on a current Arch Linux and it runs into the same problem for python2.

Here is a gdb on the coredump:

[ben@voyagerS9:~]% coredumpctl gdb 109495                                                                             [0]
Hint: You are currently not seeing messages from other users and the system.
      Users in groups 'adm', 'systemd-journal', 'wheel' can see all messages.
      Pass -q to turn off this notice.
           PID: 109495 (python2)
           UID: 1000 (ben)
           GID: 1000 (ben)
        Signal: 6 (ABRT)
     Timestamp: Sat 2019-09-21 15:25:53 CEST (1min 18s ago)
  Command Line: python2 runtests.py --coverage --no-build
    Executable: /usr/bin/python2.7
 Control Group: /user.slice/user-1000.slice/session-1.scope
          Unit: session-1.scope
         Slice: user-1000.slice
       Session: 1
     Owner UID: 1000 (ben)
       Boot ID: 8434a91c20994d54a53987ede65f99a2
    Machine ID: 3e1ebaf3deb342d6bef47fa06bba3e1c
      Hostname: voyagerS9
       Storage: /var/lib/systemd/coredump/core.python2.1000.8434a91c20994d54a53987ede65f99a2.109495.1569072353000000000000.lz4
       Message: Process 109495 (python2) of user 1000 dumped core.
                
                Stack trace of thread 109495:
                #0  0x00007f98df010755 raise (libc.so.6)
                #1  0x00007f98deffb851 abort (libc.so.6)
                #2  0x00007f98df052a38 __libc_message (libc.so.6)
                #3  0x00007f98df05925a malloc_printerr (libc.so.6)
                #4  0x00007f98df05ac50 _int_free (libc.so.6)
                #5  0x00007f98dee9af3c dict_dealloc.lto_priv.278 (libpython2.7.so.1.0)
                #6  0x00007f98dee9af3c dict_dealloc.lto_priv.278 (libpython2.7.so.1.0)
                #7  0x00007f98deee4f1a _PyImport_Fini (libpython2.7.so.1.0)
                #8  0x00007f98deee3f0e Py_Finalize (libpython2.7.so.1.0)
                #9  0x00007f98deee25f9 Py_Exit (libpython2.7.so.1.0)
                #10 0x00007f98deee30d2 handle_system_exit (libpython2.7.so.1.0)
                #11 0x00007f98deee3137 PyErr_PrintEx (libpython2.7.so.1.0)
                #12 0x00007f98deddcb30 PyRun_SimpleFileExFlags.cold.58 (libpython2.7.so.1.0)
                #13 0x00007f98deede9aa Py_Main (libpython2.7.so.1.0)
                #14 0x00007f98deffcee3 __libc_start_main (libc.so.6)
                #15 0x0000563be85a405e _start (python2.7)
                
                Stack trace of thread 109498:
                #0  0x00007f98de5107ee futex_wait (libgomp.so.1)
                #1  0x00007f98de50ded8 gomp_simple_barrier_wait (libgomp.so.1)
                #2  0x00007f98df1a257f start_thread (libpthread.so.0)
                #3  0x00007f98df0d20e3 __clone (libc.so.6)
                
                Stack trace of thread 109500:
                #0  0x00007f98de5107ee futex_wait (libgomp.so.1)
                #1  0x00007f98de50ded8 gomp_simple_barrier_wait (libgomp.so.1)
                #2  0x00007f98df1a257f start_thread (libpthread.so.0)
                #3  0x00007f98df0d20e3 __clone (libc.so.6)
                
                Stack trace of thread 109499:
                #0  0x00007f98de5107ee futex_wait (libgomp.so.1)
                #1  0x00007f98de50ded8 gomp_simple_barrier_wait (libgomp.so.1)
                #2  0x00007f98df1a257f start_thread (libpthread.so.0)
                #3  0x00007f98df0d20e3 __clone (libc.so.6)

GNU gdb (GDB) 8.3
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/bin/python2.7...
(No debugging symbols found in /usr/bin/python2.7)

warning: core file may not match specified executable file.
[New LWP 109495]
[New LWP 109498]
[New LWP 109500]
[New LWP 109499]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
Core was generated by `python2 runtests.py --coverage --no-build'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007f98df010755 in raise () from /usr/lib/libc.so.6
[Current thread is 1 (Thread 0x7f98dec405c0 (LWP 109495))]
(gdb) bt full
#0  0x00007f98df010755 in raise () from /usr/lib/libc.so.6
No symbol table info available.
#1  0x00007f98deffb851 in abort () from /usr/lib/libc.so.6
No symbol table info available.
#2  0x00007f98df052a38 in __libc_message () from /usr/lib/libc.so.6
No symbol table info available.
#3  0x00007f98df05925a in malloc_printerr () from /usr/lib/libc.so.6
No symbol table info available.
#4  0x00007f98df05ac50 in _int_free () from /usr/lib/libc.so.6
No symbol table info available.
#5  0x00007f98dee9af3c in dict_dealloc.lto_priv () from /usr/lib/libpython2.7.so.1.0
No symbol table info available.
#6  0x00007f98dee9af3c in dict_dealloc.lto_priv () from /usr/lib/libpython2.7.so.1.0
No symbol table info available.
#7  0x00007f98deee4f1a in _PyImport_Fini () from /usr/lib/libpython2.7.so.1.0
No symbol table info available.
#8  0x00007f98deee3f0e in Py_Finalize () from /usr/lib/libpython2.7.so.1.0
No symbol table info available.
#9  0x00007f98deee25f9 in Py_Exit () from /usr/lib/libpython2.7.so.1.0
No symbol table info available.
#10 0x00007f98deee30d2 in handle_system_exit () from /usr/lib/libpython2.7.so.1.0
No symbol table info available.
#11 0x00007f98deee3137 in PyErr_PrintEx () from /usr/lib/libpython2.7.so.1.0
No symbol table info available.
#12 0x00007f98deddcb30 in PyRun_SimpleFileExFlags.cold.58 () from /usr/lib/libpython2.7.so.1.0
No symbol table info available.
#13 0x00007f98deede9aa in Py_Main () from /usr/lib/libpython2.7.so.1.0
No symbol table info available.
#14 0x00007f98deffcee3 in __libc_start_main () from /usr/lib/libc.so.6
No symbol table info available.
#15 0x0000563be85a405e in _start ()
No symbol table info available.

And I think this is the relevant valgrind output:

[ben@voyagerS9:~]% valgrind python2 -c 'import slycot'

...
==109555== Invalid free() / delete / delete[] / realloc()
==109555==    at 0x48399AB: free (vg_replace_malloc.c:530)
==109555==    by 0x4BBBF3B: dict_dealloc.lto_priv.278 (in /usr/lib/libpython2.7.so.1.0)
==109555==    by 0x4BBBF3B: dict_dealloc.lto_priv.278 (in /usr/lib/libpython2.7.so.1.0)
==109555==    by 0x4C05F19: _PyImport_Fini (in /usr/lib/libpython2.7.so.1.0)
==109555==    by 0x4C04F0D: Py_Finalize (in /usr/lib/libpython2.7.so.1.0)
==109555==    by 0x4BFF74A: Py_Main (in /usr/lib/libpython2.7.so.1.0)
==109555==    by 0x4915EE2: (below main) (in /usr/lib/libc-2.29.so)
==109555==  Address 0x52adfd0 is in a rw- anonymous segment
...

Different allocation libs (libtbbmalloc, libtcmalloc* yield different error messages for the invalid free(), but fail the same.

Any tips how to debug this? I am now looking into getting the correct packages for the debuginfo.

[bnavigator]

Here is the the coredump with debuginfo on OpenSUSE:

          PID: 1541 (python2)
           UID: 399 (399)
           GID: 399 (399)
        Signal: 6 (ABRT)
     Timestamp: Sat 2019-09-21 17:48:54 CEST (12min ago)
  Command Line: python2 runtests.py --coverage --no-build --verbose
    Executable: /var/tmp/build-root/openSUSE_Tumbleweed-x86_64/usr/bin/python2.7
 Control Group: /user.slice/user-1000.slice/session-1.scope
          Unit: session-1.scope
         Slice: user-1000.slice
       Session: 1
     Owner UID: 1000 (XXX)
       Boot ID: a3933212babf41db86e7e76756b45b3e
    Machine ID: 4c67769a489a4ca689f502f35e274a39
      Hostname: XXXXXX
       Storage: /var/lib/systemd/coredump/core.python2.399.a3933212babf41db86e7e76756b45b3e.1541.1569080934000000.lz4
       Message: Process 1541 (python2) of user 399 dumped core.
                
                Stack trace of thread 1541:
                #0  0x00007fa08a6a7dd1 __GI_raise (libc.so.6)
                #1  0x00007fa08a691549 __GI_abort (libc.so.6)
                #2  0x00007fa08a6ea61f __libc_message (libc.so.6)
                #3  0x00007fa08a6f0d9c malloc_printerr (libc.so.6)
                #4  0x00007fa08a6f496e free_check (libc.so.6)
                #5  0x00007fa089ca20dc n/a (/home/abuild/rpmbuild/BUILDROOT/python-slycot-0.3.5.0-0.x86_64/usr/lib64/python2.7/site-packages/slycot/_wrapper.so)
                #6  0x00007fa08a908cfc dict_dealloc (libpython2.7.so.1.0)
                #7  0x00007fa08a908cfc dict_dealloc (libpython2.7.so.1.0)
                #8  0x00007fa08a9c59ba _PyImport_Fini (libpython2.7.so.1.0)
                #9  0x00007fa08a9c7c85 Py_Finalize (libpython2.7.so.1.0)
                #10 0x00007fa08a9c8158 Py_Exit (libpython2.7.so.1.0)
                #11 0x00007fa08a9c808a handle_system_exit (libpython2.7.so.1.0)
                #12 0x00007fa08a9c7e16 PyErr_PrintEx (libpython2.7.so.1.0)
                #13 0x00007fa08a8b7d26 PyRun_SimpleFileExFlags (libpython2.7.so.1.0)
                #14 0x00007fa08a9cccaf Py_Main (libpython2.7.so.1.0)
                #15 0x00007fa08a692bcb __libc_start_main (libc.so.6)
                #16 0x000055996de0208a _start (/usr/bin/python2.7)
                
                Stack trace of thread 1543:
                #0  0x00007fa08a659e05 futex_wait_cancelable (libpthread.so.0)
                #1  0x00007fa0891631fb blas_thread_server (libopenblas_pthreads.so.0)
                #2  0x00007fa08a653faa start_thread (libpthread.so.0)
                #3  0x00007fa08a76973f __clone (libc.so.6)
                
                Stack trace of thread 1542:
                #0  0x00007fa08a659e05 futex_wait_cancelable (libpthread.so.0)
                #1  0x00007fa0891631fb blas_thread_server (libopenblas_pthreads.so.0)
                #2  0x00007fa08a653faa start_thread (libpthread.so.0)
                #3  0x00007fa08a76973f __clone (libc.so.6)
                
                Stack trace of thread 1544:
                #0  0x00007fa08a659e05 futex_wait_cancelable (libpthread.so.0)
                #1  0x00007fa0891631fb blas_thread_server (libopenblas_pthreads.so.0)
                #2  0x00007fa08a653faa start_thread (libpthread.so.0)
                #3  0x00007fa08a76973f __clone (libc.so.6)

And the full bdg backtrace:
python2-slycot-gdb.txt

[bnavigator]

Found it!

CMake was using the wrong f2py. This is both true for Arch Linux and OpenSUSE. When both python2-numpy and python(3)-numpy are installed, CMake keeps finding /usr/bin/f2py which is the Python 3 version.

By forcing /usr/bin/f2py2 in the python2 build, everything builds fine
python2 setup.py build -- -DF2PY_EXECUTABLE=/usr/bin/f2py2

Not much to fix in in Slycot, I guess, but I will file a bug in scikit-build.