np.asarray - unexpected behaviour

[deviganesan-coder]

When I try to do np.asarray of a list of arrays, it results in different behaviour depending on the dimensions of the arrays. When the list of arrays is of dimensions like (n,n) and (n,) it results in error.

Reproducing code example:

import numpy as np
a1 = [array([[-0.00491985,  0.00491965],
        [-0.00334969,  0.00334955],
        [-0.00136081,  0.00136076]], dtype=float32),
 array([-0.00104678,  0.00104674], dtype=float32)]

a2 = [array([[-0.00334969,  0.00334955],
        [-0.00136081,  0.00136076]], dtype=float32),
 array([-0.00104678,  0.00104674], dtype=float32)]

print(np.asarray(a1))   #works fine
print(np.asarray(a2))  #throws error
print(np.asarray([a.reshape(-1, 2) for a in a2]))  #works fine

Error message:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-51-3060768e9016> in <module>()
----> 1 np.asarray(a2)

/home/devi/.local/lib/python3.5/site-packages/numpy/core/numeric.py in asarray(a, dtype, order)
    536 
    537     """
--> 538     return array(a, dtype, copy=False, order=order)
    539 
    540 

ValueError: could not broadcast input array from shape (2,2) into shape (2)

Numpy/Python version information:

('1.16.3', '2.7.12 (default, Nov 12 2018, 14:36:49) \n[GCC 5.4.0 20160609]')

[zjpoh]

Another example that raises the same error.

np.asarray([np.ones((n, m)), np.ones((n))])

I suspect the error is because when you have a list of arrays which have the same first dimension of the shape, then the np.asarray is trying to make it to a single array instead of an array of array.

Similarly error also occurs for multi-dimensional arrays.

I can take a look at this if nobody else is working on it.

[eric-wieser]

I can take a look at this if nobody else is working on it

Feel free too, but I'll warn you that this has almost certainly come up before, and is rather tricky to fix.

[zjpoh]

😱I'll take a look and see if I can figure out something

[leonardbinet]

I also have unexpected behavior with csr matrices:

import numpy as np
from scipy.sparse import csr_matrix

a = np.array([[1, 0, 2, 2], [1, 4, 2, 4]])
m = csr_matrix(a)

m.shape
Out: (2, 4)

m.todense()
Out:
matrix([[1, 0, 2, 2],
        [1, 4, 2, 4]])

m_2 = np.asarray(m)
# weird shape
m_2.shape
Out: ()

# whereas it seems to be conserved somehow
m_2
Out:
array(<2x4 sparse matrix of type '<type 'numpy.int64'>'
	with 7 stored elements in Compressed Sparse Row format>, dtype=object)


m_2.todense()
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-12-3ca00cd97e3f> in <module>()
----> 1 m_2.todense()

AttributeError: 'numpy.ndarray' object has no attribute 'todense'

versions:
1.17.1 3.7.2 (default, Feb 12 2019, 08:16:38)
[Clang 10.0.0 (clang-1000.11.45.5)]

 python: 3.7.2 
 scipy: 1.3.1

[zjpoh]

I'm driving into this. Do you know what should be the expected behavior? Should it return an array of arrays? Or should it reshape and return a single np arrays.

[rgommers]

The sparse matrix thing is a different issue.

Do you know what should be the expected behavior? Should it return an array of arrays? Or should it reshape and return a single np arrays.

In [21]: np.asarray([np.ones((5, 3)), np.ones((4))])                               
Out[21]: 
array([array([[1., 1., 1.],
       [1., 1., 1.],
       [1., 1., 1.],
       [1., 1., 1.],
       [1., 1., 1.]]),
       array([1., 1., 1., 1.])], dtype=object)

In [22]: np.asarray([np.ones((5, 3)), np.ones((3))])                               
Out[22]: 
array([array([[1., 1., 1.],
       [1., 1., 1.],
       [1., 1., 1.],
       [1., 1., 1.],
       [1., 1., 1.]]),
       array([1., 1., 1.])], dtype=object)

In [23]: np.asarray([np.ones((3, 5)), np.ones((3))])                               
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-23-596edbc3970a> in <module>
----> 1 np.asarray([np.ones((3, 5)), np.ones((3))])

~/anaconda3/lib/python3.7/site-packages/numpy/core/numeric.py in asarray(a, dtype, order)
    536 
    537     """
--> 538     return array(a, dtype, copy=False, order=order)
    539 
    540 

ValueError: could not broadcast input array from shape (3,5) into shape (3)

The consistent thing for this particular issue would be to not get the ValueError but make it return the same object-array-of-arrays as the other case.

Of course such behavior is probably still not what the user intended ...... if we'd want to change it, that should probably raise an exception. I don't really know, it's tricky.

[WarrenWeckesser]

FYI: The somewhat surprising automatic conversion of the asarray arguments to an object array when the arguments have non-conformable shapes was discussed in #5303.

[zjpoh]

@rgommers Thanks. I agree on your point on consistency. But I also agree that it is probably not the intended behavior and making it consistent might hide the problem.

Yeap. The sparse matrix should be a separate issue. Let me move that over.

@WarrenWeckesser Thanks for pointing that out. This is a really tricky issue.

[eric-wieser]

@zaraasghar, your issue is completely unrelated, and looks like a user error. I'd recommend asking on stack overflow.

[seberg]

All of the examples now correctly throw an error.