Call to scipy.sparse.coo_matrix fails when passed Cython typed memoryviews

[jbweston]

Under Python 3, when calling scipy.sparse.coo_matrix from a Cython extension module and passing Cython typed memoryviews as arguments, the call raises a ValueError. What follows is an example of the relevant calling code:

import scipy.sparse as sp

...  # snip

cdef signed int [:] to_off
cdef signed int [:] to_off
cdef signed int [:, :] rows_cols
cdef complex [:] data

...  # snip -- assign the memoryviews to valid buffers

return sp.coo_matrix((data[:], rows_cols[:, :]), shape=(to_off[-1], from_off[-1]))

Produces the following traceback

Traceback (most recent call last):
  File "/home/jbw/data/src/kwant/python3-env/lib/python3.4/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/jbw/data/src/kwant/python3-env/lib/python3.4/site-packages/kwant/tests/test_system.py", line 33, in test_hamiltonian_submatrix
    mat = sys2.hamiltonian_submatrix(sparse=True)
  File "kwant/_system.pyx", line 313, in kwant._system.hamiltonian_submatrix (kwant/_system.c:8029)
  File "kwant/_system.pyx", line 151, in kwant._system.make_sparse_full (kwant/_system.c:4482)
  File "/home/jbw/data/src/kwant/python3-env/lib/python3.4/site-packages/scipy/sparse/coo.py", line 120, in __init__
    if isshape(arg1):
  File "/home/jbw/data/src/kwant/python3-env/lib/python3.4/site-packages/scipy/sparse/sputils.py", line 212, in isshape
    if isintlike(M) and isintlike(N):
  File "/home/jbw/data/src/kwant/python3-env/lib/python3.4/site-packages/scipy/sparse/sputils.py", line 195, in isintlike
    if int(x) == x:
ValueError: invalid literal for int() with base 10: b'\x00\x00\x00\x00\x00\x00\xf0?\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00@\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xc0\x00\x00\x00

The problem has been traced to scipy.sparse.sputils.isintlike. There is a call to int inside a try block where only TypeError is caught and handled. In Python 2 calling int on a memoryview produced a TypeError, but in Python 3 it produces a ValueError.

The problem has been reproduced with the current development version of Scipy: 0.17.0.dev0+fa26145 and Python 3.4.3+.

[jbweston]

It should be noted that the above does work under Python 2.7.

[ewmoore]

You can work around this by converting your memoryviews to numpy arrays. They'll share memory, so this shouldn't be a big deal. Additionally, if the memoryviews were detected as sequences, they'll be converted to arrays anyway.

I kinda suspect that you're going to find issues like this when feeding memoryviews to arbitrary functions. Most things that expect numpy arrays call asarray first, but not everything does and nothing is expecting to be passed a memoryview.