Buffer dtype mismatch, expected 'ITYPE_t' but got 'long'

[shervinf6687]

I am trying to get connected components in an adjacency matrix which is quite big but sparse. However when I run the code, I get this error:
ValueError: Buffer dtype mismatch, expected 'ITYPE_t' but got 'long'
Exception ignored in: 'scipy.sparse.csgraph._traversal._connected_components_undirected'
ValueError: Buffer dtype mismatch, expected 'ITYPE_t' but got 'long'

When I run the code for a smaller matrix everything works fine.

Reproducing Code Example

from scipy import sparse
adj_matrix = sparse.load_npz(adj_matrix_address).astype(int)

# adj_matrix  only has zeros and ones
# <3556401x3556401 sparse matrix of type '<class 'numpy.int64'>'
#	with 6917339542 stored elements in Compressed Sparse Row format>

n_cluster, cluster_labels = sparse.csgraph.connected_components(csgraph=adj_matrix, directed=False, return_labels=True)

Error message

ValueError: Buffer dtype mismatch, expected 'ITYPE_t' but got 'long'
Exception ignored in: 'scipy.sparse.csgraph._traversal._connected_components_undirected'
ValueError: Buffer dtype mismatch, expected 'ITYPE_t' but got 'long'

SciPy/NumPy/Python version and system information

lapack_mkl_info:
  NOT AVAILABLE
openblas_lapack_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/usr/local/lib']
    language = c
    define_macros = [('HAVE_CBLAS', None)]
    runtime_library_dirs = ['/usr/local/lib']
lapack_opt_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/usr/local/lib']
    language = c
    define_macros = [('HAVE_CBLAS', None)]
    runtime_library_dirs = ['/usr/local/lib']
blas_mkl_info:
  NOT AVAILABLE
blis_info:
  NOT AVAILABLE
openblas_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/usr/local/lib']
    language = c
    define_macros = [('HAVE_CBLAS', None)]
    runtime_library_dirs = ['/usr/local/lib']
blas_opt_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/usr/local/lib']
    language = c
    define_macros = [('HAVE_CBLAS', None)]
    runtime_library_dirs = ['/usr/local/lib']

[dschmitz89]

Could you upload your matrix as a .npz file? Otherwise this will not be possible to debug.

[perimosocordiae]

I don't think uploading the matrix will be necessary here, actually. Instead, could you print adj_matrix.indices.dtype and also adj_matrix.indptr.dtype?

[anntzer]

I suspect this is a fallout of #18509 (or something related); a simple repro is

import numpy as np; from scipy import sparse as sp
print(sp.csgraph.connected_components(sp.coo_array((np.array([1]), np.array([[0], [1]])), (2, 2))))

which prints (1, array([0, 0], dtype=int32)) with scipy 1.10.1 and errors with

ValueError: Buffer dtype mismatch, expected 'ITYPE_t' but got 'long'
Exception ignored in: 'scipy.sparse.csgraph._traversal._connected_components_undirected'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ValueError: Buffer dtype mismatch, expected 'ITYPE_t' but got 'long'
(0, array([-9999, -9999], dtype=int32))

(as above) with scipy 1.11.1.
In particular, note the "Exception ignored", which is pretty awkward: even though the exception gets printed to stdout, the error is then discarded and an incorrect value is actually returned.

[perimosocordiae]

Thanks for the reproducer, @anntzer ! This definitely looks like fallout from #18509, similar to the recently-fixed #18603.

Looking at the Cython code in question, we may be able to fix this by using a fused index type in _traversal.pyx, as in

scipy/scipy/sparse/csgraph/_traversal.pyx

Lines 647 to 650 in ddee711

	cdef int _connected_components_directed(
	np.ndarray[ITYPE_t, ndim=1, mode='c'] indices,
	np.ndarray[ITYPE_t, ndim=1, mode='c'] indptr,
	np.ndarray[ITYPE_t, ndim=1, mode='c'] labels) noexcept:

There's a precedent for this in other csgraph Cython code:

scipy/scipy/sparse/csgraph/_reordering.pyx

Lines 102 to 103 in 3c89445

	def _reverse_cuthill_mckee(np.ndarray[int32_or_int64, ndim=1, mode="c"] ind,
	np.ndarray[int32_or_int64, ndim=1, mode="c"] ptr,

The other option is to do what we did in #18644 and downcast the index arrays before calling the Cython code, but I'd prefer to try the fused type first.