Support for case where number of stored elements in one of the matrices is greater than the range of int32

[DeepakSaini119]

Hi, I want to do a dot product of two sparse matrices and one of the matrix has ~4bil stored elements. I get this error:

MKL interface is <class 'numpy.int32'> and cannot hold matrix <47494534x2619400 sparse matrix of type '<class 'numpy.float32'>'
	with 4328780604 stored elements in Compressed Sparse Row format

It seems to me it is because the number of stored elements are out of the range of int32. Am i correct? And if so, is there a way to do this using sparse_dot_mkl. TIA.

[asistradition]

Yes, that's correct - the LP64 interface for MKL is int32 and can't deal with more than 2^31 − 1 elements in the matrix.

The good news is that there's an ILP64 interface for MKL that uses int64 and won't have this problem. The bad news is that I was never able to get it to work - there's a function and an environment variable that can be used to change the interface but neither worked for me. If you can switch the interface everything should just work, but I don't know how to do that.

If I get time I'll look into linking directly to the ilp64 libraries instead of trying to go through the main interface, but that's enough work that I haven't done it yet. Sorry.

[asistradition]

I looked into this a little more - the MKL that's distributed with anaconda doesn't appear to support the int64 interface and I don't know why. I installed the intel channel mkl and it worked fine.

conda install -c intel mkl

$ export MKL_INTERFACE_LAYER=ILP64
$ python
Python 3.8.3 (default, May 19 2020, 18:47:26) 
>>> from sparse_dot_mkl._mkl_interface import MKL
>>> MKL.MKL_INT_NUMPY
<class 'numpy.int64'>
>>> quit()

$ export MKL_INTERFACE_LAYER=LP64
$ python
Python 3.8.3 (default, May 19 2020, 18:47:26) 
>>> from sparse_dot_mkl._mkl_interface import MKL
>>> MKL.MKL_INT_NUMPY
<class 'numpy.int32'>
>>> 

I think this should resolve your problem.

[chuckzzzz]

Just wanna throw a quick comment.

If everything fails and you cannot override the 2^31-1 limit maximum elements, you could just breakup matrix multiplication into quadrants to kinda go around it. This was what I did with a 90000X90000 matrix. Clearly this would sacrifice runtime, but I think it's worth it.

p.s.: This package is very useful so THANK YOU!

[ivirshup]

I looked into this a little more - the MKL that's distributed with anaconda doesn't appear to support the int64 interface and I don't know why.

I believe that this works now. I've tried the intel, conda-forge, and defaults channel and all seem to support 64 bit integers.

[yCobanoglu]

It does not work. Don't use conda if you want to use MKL_INTERFACE_LAYER=ILP64. The env variable has no effect according to the code snippet @asistradition.