ENH: Improved performance of PyArray_FromAny for sequences of array-like

[superbobry]

Prior to this commit

np.array([array_like])

would recursively copy each element of array_like. This is due to the fact that
setArrayFromSequence only special-cased lists of NumPy arrays, any other
object was treated as a sequence even if it supported buffer or __array*__
interfaces. See tensorflow/tensorflow#27692 for details.

The commit generalizes the special-case in setArrayFromSequence to any
array-like, i.e. a buffer or an object with __array__, __array_interface__
__array_struct__.

[mattip]

Running

python -mtimeit -s "import numpy as np; b=memoryview(np.arange(1000))" \
       "a = np.array([b] * 100)"

shows a marked improvement. Before

100 loops, best of 3: 2.6 msec per loop

After

10000 loops, best of 3: 71.9 usec per loop

[mattip]

For reviewers: the original code had a check clause at this point if (tmp != Py_NotImplemented){ } which has been folded into the logic below instead

[mattip]

LGTM. Maybe could add a benchmark to benchmarks/benchmarks/bench_core.Core.time*.

[eric-wieser]

I worry that this needs to handle broken ctypes buffer formats in the same way we do elsewhere. Will try to look at the diff later

[superbobry]

I was also thinking of addressing #8562 but I hesitated because buffer interface allows for ~free shape queries whereas __array__ potentially allocates a new array which seems wasteful.

@mattip I've added a benchmark, could you have another look please?

[mattip]

I was also thinking of addressing #8562

Great, but that should be a follow-on PR, not part of this one, unless it is a one-line fix

[mattip]

@eric-wieser since this PR reuses existing code, could we push off possible broken ctypes PEP 3118 handling to a different issue/PR? This seems like a clear win for more common memoryview and __array__ use cases

[eric-wieser]

@mattip: At a cursory glance, I was worried this introduced a regression in the following case:

class PaddedStruct(ctypes.Structure):
    _fields_ = [('a', ctypes.c_uint8), ('b', ctypes.c_uint16)]

data = (PaddedStruct * 4)()
np.array(data)  # ok
np.array(list(data))
# ValueError: invalid literal for int() with base 10: b'\x00\x00\x00\x00'

Testing seems to show that the error is produced both before and after this patch, so there's no need to worry about it here.

[mattip]

I created issue #13628 about np.array(list(ctypes-struct)) which does not need the padding to fail.

This looks ready to merge, any more comments?

[mattip]

Thanks @superbobry

[mattip]

This seems to have broken pandas, where import numpy as np, pandas as pd; np.array([pd.DataFrame()]) used to work and now raises ValueError: could not broadcast input array from shape (0,0) into shape (0)