Crash when using AB09ND with bfsqrt, issue is incorrect dimension for iwork

[mccullerlp]

I've had consistent crashing when trying to use the bfsqrt method with AB09ND. I switched on debug mode, -fcheck=all and such. I was able to track it down after also poking around using valgrind. For future reference, it is nice to run test suites that fail with memory checking and gdb to poke around inside the stack traces. This is much better than just what -fcheck=all does.

PYTHONMALLOC=malloc valgrind --vgdb-error=1 python3 -m pytest <args>

see https://developers.redhat.com/articles/2021/11/01/debug-memory-errors-valgrind-and-gdb#discover_uninitialized_values for more details.

Anyway, for the bug fix. In the wrapper for AB09ND,

    integer intent(hide,cache),dimension(MAX(m,p)) :: iwork

should become

    integer intent(hide,cache),dimension(MAX(1,2*n)),depend(n) :: iwork

which makes it consistent with the requirements in the slicot source file, and also appears to fix the issue.

[roryyorke]

Hi - thanks for that.

See gh-243 , linked from gh-242 that you filed last week. This has the same expression for iwork, but doesn't have depend(n). From https://numpy.org/doc/stable/f2py/signature-file.html#signature-files-syntax, f2py infers dependence, and depend() is used to override its inference.

Do you get crashes without depend(n) ?

[mccullerlp]

I will check this soon. My work on this is really sporadic right now. Will clean up that pull request and have a couple more wrapped functions (SB08CD and SB08DD) hopefully within a month or so.

[roryyorke]

python-control/python-control#1177 may be related. I can reproduce a segfault with Slycot 2158957 , which has gh-243 in, using the script below. Partial valgrind report at the end.

import sys

import numpy as np

from slycot import ab09nd

a = np.array([[0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,-0.04761905],
[1.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,-0.0952381],
[0.,1.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,-0.14285714],
[0.,0.,1.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,-0.19047619],
[0.,0.,0.,1.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,-0.23809524],
[0.,0.,0.,0.,1.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,-0.28571429],
[0.,0.,0.,0.,0.,1.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,-0.33333333],
[0.,0.,0.,0.,0.,0.,1.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,-0.38095238],
[0.,0.,0.,0.,0.,0.,0.,1.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,-0.42857143],
[0.,0.,0.,0.,0.,0.,0.,0.,1.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,-0.47619048],
[0.,0.,0.,0.,0.,0.,0.,0.,0.,1.,0.,0.,0.,0.,0.,0.,0.,0.,0.,-0.52380952],
[0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,1.,0.,0.,0.,0.,0.,0.,0.,0.,-0.57142857],
[0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,1.,0.,0.,0.,0.,0.,0.,0.,-0.61904762],
[0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,1.,0.,0.,0.,0.,0.,0.,-0.66666667],
[0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,1.,0.,0.,0.,0.,0.,-0.71428571],
[0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,1.,0.,0.,0.,0.,-0.76190476],
[0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,1.,0.,0.,0.,-0.80952381],
[0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,1.,0.,0.,-0.85714286],
[0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,1.,0.,-0.9047619],
[0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,1.,-0.95238095]])

b = np.array([[1]] + [[0] for i in range(19)])
c = np.array([[1] + [0]*19])
d = 0

ab09nd(dico='C',
       job='B',
       equil='N',
       n=20,
       m=1,
       p=1,
       A=a,
       B=b,
       C=c,
       D=d,
       alpha=0,
       nr=10,
       tol1=0.0,
       tol2=0.0)

Partial valgrind output; this is valgrind --log-file=ex8.log python ex8.py, ex8.py being the above.

==129980== Memcheck, a memory error detector
==129980== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==129980== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==129980== Command: python ex8.py
==129980== Parent PID: 29703
==129980== 
==129980== Conditional jump or move depends on uninitialised value(s)
==129980==    at 0x38FC3B: PyUnicode_Decode (in /home/rory/miniforge3/envs/slycot-dev/bin/python3.12)
==129980==    by 0x381B8E: unicode_new (in /home/rory/miniforge3/envs/slycot-dev/bin/python3.12)
==129980==    by 0x31697C: _PyObject_MakeTpCall (in /home/rory/miniforge3/envs/slycot-dev/bin/python3.12)
==129980==    by 0x21B1F2: _PyEval_EvalFrameDefault.cold (in /home/rory/miniforge3/envs/slycot-dev/bin/python3.12)
==129980==    by 0x31A829: _PyObject_FastCallDictTstate (in /home/rory/miniforge3/envs/slycot-dev/bin/python3.12)
==129980==    by 0x34923E: slot_tp_init (in /home/rory/miniforge3/envs/slycot-dev/bin/python3.12)
==129980==    by 0x316A5B: _PyObject_MakeTpCall (in /home/rory/miniforge3/envs/slycot-dev/bin/python3.12)
==129980==    by 0x21B1F2: _PyEval_EvalFrameDefault.cold (in /home/rory/miniforge3/envs/slycot-dev/bin/python3.12)
==129980==    by 0x3E055D: PyEval_EvalCode (in /home/rory/miniforge3/envs/slycot-dev/bin/python3.12)
==129980==    by 0x3FAD9C: builtin_exec (in /home/rory/miniforge3/envs/slycot-dev/bin/python3.12)
==129980==    by 0x32DC15: cfunction_vectorcall_FASTCALL_KEYWORDS (in /home/rory/miniforge3/envs/slycot-dev/bin/python3.12)
==129980==    by 0x21BFFF: _PyEval_EvalFrameDefault.cold (in /home/rory/miniforge3/envs/slycot-dev/bin/python3.12)
==129980== 
==129980== Invalid write of size 4
==129980==    at 0x60D111A: dgetf2_k (in /home/rory/miniforge3/envs/slycot-dev/lib/libopenblasp-r0.3.24.so)
==129980==    by 0x5E88D5D: dgetrf_ (in /home/rory/miniforge3/envs/slycot-dev/lib/libopenblasp-r0.3.24.so)
==129980==    by 0x23D712D5: ab09dd_ (in /home/rory/src/slycot/slycot/_wrapper.cpython-312-x86_64-linux-gnu.so)
==129980==    by 0x23D70E87: ab09bx_ (in /home/rory/src/slycot/slycot/_wrapper.cpython-312-x86_64-linux-gnu.so)
==129980==    by 0x23D72810: ab09nd_ (in /home/rory/src/slycot/slycot/_wrapper.cpython-312-x86_64-linux-gnu.so)
==129980==    by 0x23D374DA: f2py_rout__wrapper_ab09nd (in /home/rory/src/slycot/slycot/_wrapper.cpython-312-x86_64-linux-gnu.so)
==129980==    by 0x316A8A: _PyObject_MakeTpCall (in /home/rory/miniforge3/envs/slycot-dev/bin/python3.12)
==129980==    by 0x21B1F2: _PyEval_EvalFrameDefault.cold (in /home/rory/miniforge3/envs/slycot-dev/bin/python3.12)
==129980==    by 0x3E055D: PyEval_EvalCode (in /home/rory/miniforge3/envs/slycot-dev/bin/python3.12)
==129980==    by 0x404019: run_eval_code_obj (in /home/rory/miniforge3/envs/slycot-dev/bin/python3.12)
==129980==    by 0x3FF1BA: run_mod (in /home/rory/miniforge3/envs/slycot-dev/bin/python3.12)
==129980==    by 0x415B1F: pyrun_file (in /home/rory/miniforge3/envs/slycot-dev/bin/python3.12)
==129980==  Address 0x23adccb4 is 0 bytes after a block of size 4 alloc'd
==129980==    at 0x483B7F3: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==129980==    by 0x587241E: PyDataMem_UserNEW (in /home/rory/miniforge3/envs/slycot-dev/lib/python3.12/site-packages/numpy/core/_multiarray_umath.cpython-312-x86_64-linux-gnu.so)
==129980==    by 0x58905C7: PyArray_NewFromDescr_int (in /home/rory/miniforge3/envs/slycot-dev/lib/python3.12/site-packages/numpy/core/_multiarray_umath.cpython-312-x86_64-linux-gnu.so)
==129980==    by 0x589092F: PyArray_NewFromDescr (in /home/rory/miniforge3/envs/slycot-dev/lib/python3.12/site-packages/numpy/core/_multiarray_umath.cpython-312-x86_64-linux-gnu.so)
==129980==    by 0x595C8C9: try_trivial_single_output_loop (in /home/rory/miniforge3/envs/slycot-dev/lib/python3.12/site-packages/numpy/core/_multiarray_umath.cpython-312-x86_64-linux-gnu.so)
==129980==    by 0x596529D: ufunc_generic_fastcall (in /home/rory/miniforge3/envs/slycot-dev/lib/python3.12/site-packages/numpy/core/_multiarray_umath.cpython-312-x86_64-linux-gnu.so)
==129980==    by 0x32D96E: PyObject_Vectorcall (in /home/rory/miniforge3/envs/slycot-dev/bin/python3.12)
==129980==    by 0x21B1F2: _PyEval_EvalFrameDefault.cold (in /home/rory/miniforge3/envs/slycot-dev/bin/python3.12)
==129980==    by 0x31A8C3: _PyObject_FastCallDictTstate (in /home/rory/miniforge3/envs/slycot-dev/bin/python3.12)
==129980==    by 0x34923E: slot_tp_init (in /home/rory/miniforge3/envs/slycot-dev/bin/python3.12)
==129980==    by 0x316A5B: _PyObject_MakeTpCall (in /home/rory/miniforge3/envs/slycot-dev/bin/python3.12)
==129980==    by 0x21B1F2: _PyEval_EvalFrameDefault.cold (in /home/rory/miniforge3/envs/slycot-dev/bin/python3.12)
==129980== 
==129980== Invalid read of size 4
==129980==    at 0x60D0FDC: dgetf2_k (in /home/rory/miniforge3/envs/slycot-dev/lib/libopenblasp-r0.3.24.so)
==129980==    by 0x5E88D5D: dgetrf_ (in /home/rory/miniforge3/envs/slycot-dev/lib/libopenblasp-r0.3.24.so)
==129980==    by 0x23D712D5: ab09dd_ (in /home/rory/src/slycot/slycot/_wrapper.cpython-312-x86_64-linux-gnu.so)
==129980==    by 0x23D70E87: ab09bx_ (in /home/rory/src/slycot/slycot/_wrapper.cpython-312-x86_64-linux-gnu.so)
==129980==    by 0x23D72810: ab09nd_ (in /home/rory/src/slycot/slycot/_wrapper.cpython-312-x86_64-linux-gnu.so)
==129980==    by 0x23D374DA: f2py_rout__wrapper_ab09nd (in /home/rory/src/slycot/slycot/_wrapper.cpython-312-x86_64-linux-gnu.so)
==129980==    by 0x316A8A: _PyObject_MakeTpCall (in /home/rory/miniforge3/envs/slycot-dev/bin/python3.12)
==129980==    by 0x21B1F2: _PyEval_EvalFrameDefault.cold (in /home/rory/miniforge3/envs/slycot-dev/bin/python3.12)
==129980==    by 0x3E055D: PyEval_EvalCode (in /home/rory/miniforge3/envs/slycot-dev/bin/python3.12)
==129980==    by 0x404019: run_eval_code_obj (in /home/rory/miniforge3/envs/slycot-dev/bin/python3.12)
==129980==    by 0x3FF1BA: run_mod (in /home/rory/miniforge3/envs/slycot-dev/bin/python3.12)
==129980==    by 0x415B1F: pyrun_file (in /home/rory/miniforge3/envs/slycot-dev/bin/python3.12)
==129980==  Address 0x23adccb4 is 0 bytes after a block of size 4 alloc'd
==129980==    at 0x483B7F3: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==129980==    by 0x587241E: PyDataMem_UserNEW (in /home/rory/miniforge3/envs/slycot-dev/lib/python3.12/site-packages/numpy/core/_multiarray_umath.cpython-312-x86_64-linux-gnu.so)
==129980==    by 0x58905C7: PyArray_NewFromDescr_int (in /home/rory/miniforge3/envs/slycot-dev/lib/python3.12/site-packages/numpy/core/_multiarray_umath.cpython-312-x86_64-linux-gnu.so)
==129980==    by 0x589092F: PyArray_NewFromDescr (in /home/rory/miniforge3/envs/slycot-dev/lib/python3.12/site-packages/numpy/core/_multiarray_umath.cpython-312-x86_64-linux-gnu.so)
==129980==    by 0x595C8C9: try_trivial_single_output_loop (in /home/rory/miniforge3/envs/slycot-dev/lib/python3.12/site-packages/numpy/core/_multiarray_umath.cpython-312-x86_64-linux-gnu.so)
==129980==    by 0x596529D: ufunc_generic_fastcall (in /home/rory/miniforge3/envs/slycot-dev/lib/python3.12/site-packages/numpy/core/_multiarray_umath.cpython-312-x86_64-linux-gnu.so)
==129980==    by 0x32D96E: PyObject_Vectorcall (in /home/rory/miniforge3/envs/slycot-dev/bin/python3.12)
==129980==    by 0x21B1F2: _PyEval_EvalFrameDefault.cold (in /home/rory/miniforge3/envs/slycot-dev/bin/python3.12)
==129980==    by 0x31A8C3: _PyObject_FastCallDictTstate (in /home/rory/miniforge3/envs/slycot-dev/bin/python3.12)
==129980==    by 0x34923E: slot_tp_init (in /home/rory/miniforge3/envs/slycot-dev/bin/python3.12)
==129980==    by 0x316A5B: _PyObject_MakeTpCall (in /home/rory/miniforge3/envs/slycot-dev/bin/python3.12)
==129980==    by 0x21B1F2: _PyEval_EvalFrameDefault.cold (in /home/rory/miniforge3/envs/slycot-dev/bin/python3.12)

[roryyorke]

It is fixed after all, I missed the --inplace argument to python setup.py build_ext when testing. I tried with a slightly different example (see below). With nsys=200 and nred=100, it reliably segfaults on v0.6.0, but on master 2158957 this is fine.

import sys

import numpy as np
from numpy.polynomial.polynomial import polycompanion

from slycot import ab09nd

nsys = int(sys.argv[1])
nred = int(sys.argv[2])

a = polycompanion(np.arange(1, nsys+2))
b = np.eye(nsys,1)
c = np.eye(1, nsys)
d = 0

ab09nd(dico='C',
       job='B',
       equil='N',
       n=nsys,
       m=1,
       p=1,
       A=a,
       B=b,
       C=c,
       D=d,
       alpha=0,
       nr=nred,
       tol1=0.0,
       tol2=0.0)