pythran breaks scipy on 32 bit arches (i386, armhf, armel, mipsel, powerpc)

[drew-parsons]

Debian is now configuring scipy to use pythran on all architectures. For most of our CI test arches pass fine, but there are problems with armhf. A dozen or more dependent packages failed their CI tests, typically with MemoryError. That's not such a big problem, it might just mean the specific test cases were too large. But scipy's own tests failed with pythran on armhf. An example test log testing scipy 1.8.1-5 with pythran 0.11.0+ds-3 is found at https://ci.debian.net/data/autopkgtest/testing/armhf/s/scipy/23186072/log.gz

Some of the failures are just questions of tolerance. But there were some specific pythran errors, e.g. in scipy/optimize/tests/test__numdiff.py

______________________________ test_group_columns ______________________________
/usr/lib/python3/dist-packages/scipy/optimize/tests/test__numdiff.py:29: in test_group_columns
    groups = group_columns(A, order)
        A          = <7x6 sparse matrix of type '<class 'numpy.intc'>'
	with 16 stored elements in Compressed Sparse Row format>
        groups     = array([2, 0, 1, 2, 0, 1], dtype=int32)
        groups_true = array([0, 1, 2, 0, 1, 2])
        order      = array([0, 1, 2, 3, 4, 5])
        structure  = [[1, 1, 0, 0, 0, 0], [1, 1, 1, 0, 0, 0], [0, 1, 1, 1, 0, 0], [0, 0, 1, 1, 1, 0], [0, 0, 0, 1, 1, 1], [0, 0, 0, 0, 1, 1], ...]
        transform  = <class 'scipy.sparse._csr.csr_matrix'>
/usr/lib/python3/dist-packages/scipy/optimize/_numdiff.py:266: in group_columns
    groups = group_sparse(m, n, A.indices, A.indptr)
E   TypeError: Invalid call to pythranized function `group_sparse(int, int, intc[:] (is a view), intc[:])'
E   Candidates are:
E   
E       - group_sparse(int, int, int[::], int[::])
E       - group_sparse(int, int, int32[::], int32[::])
E       - group_sparse(int, int, int[:], int[:])
E       - group_sparse(int, int, int32[:], int32[:])
        A          = <7x6 sparse matrix of type '<class 'numpy.intc'>'
	with 16 stored elements in Compressed Sparse Column format>
        m          = 7
        n          = 6
        order      = array([0, 1, 2, 3, 4, 5])

The same pythranized group_sparse error was seen in other tests, e.g. test_check_derivative, optimize/tests/test_least_squares.py (test_numerical_jac, TestDogbox.test_with_bounds)

Not sure if it should be fixed in scipy or in pythran, or if it means pythran shouldn't be used on armhf. But pythran 0.11 is passing its own tests on armhf.

In the meantime for the debian packaging I'll configure pythran to be optional for scipy on armhf.

[serge-sans-paille]

This is probably a pythran signature issue, I'll have a look. Thanks for reporting!

[serge-sans-paille]

@drew-parsons Can you test if scipy/scipy#16646 fixes the issue? Thanks!

[drew-parsons]

Incidently the problem seems to be on all 32-bit arches. i386 and armel are also affected by the "Invalid call to pythranized function" error. Also powerpc.

[drew-parsons]

scipy PR#16646 fixes the reported problems, but identifies another case needing to be fixed in signal/tests/test_max_len_seq.py

mipsel is also affected so I've updated the title to refer to all 32 bit arches