Array shape is lost during as_ctypes/as_array round trip in 1.15

[Strilanc]

It appears that, when a 2 dimensional array's first index has a length of 1, round-tripping the array through np.ctypeslib.as_ctypes and np.ctypeslib.as_array loses the shape of the array.

Here is a short repro:

import numpy as np
arr = np.array([[1, 2]])
c_arr = np.ctypeslib.as_ctypes(arr)
back = np.ctypeslib.as_array(c_arr)
np.testing.assert_equal(arr, back)

# In numpy 1.15:
# E       AssertionError: 
# E       Arrays are not equal
# E       
# E       (shapes (1, 2), (2,) mismatch)
# E        x: array([[1, 2]])
# E        y: array([1, 2])

Note that this did not occur in numpy v1.14. Another example of an array that doesn't round trip correctly is [[1], [2]]. However, [[1, 2], [3, 4]] does round trip correctly.

Note that a possibly-relevant warning which I do not understand occurs when executing the repro code:

RuntimeWarning: Item size computed from the PEP 3118 buffer format string does not match the actual item size.

[eric-wieser]

a possibly-relevant warning which I do not understand occurs

This warning is telling you that your version of python has a broken ctypes module. Unfortunately, there is no version of python yet released with a PEP3118-compliant ctypes module...

[eric-wieser]

I can't reproduce this - what is sys.version_info on your platform?

Can you show the result of print(repr(c_arr)) and of memoryview(c_arr).format?

[Strilanc]

In [2]: print(repr(c_arr))
<numpy.ctypeslib.c_long_Array_2_Array_1 object at 0x7f8f648ff950>

In [3]: memoryview(c_arr).format
Out[3]: '<l'

In [4]: import sys

In [5]: print(sys.version_info)
sys.version_info(major=3, minor=5, micro=3, releaselevel='final', serial=0)

[eric-wieser]

Well that's pretty weird. Can you also show:

• memoryview(c_arr).itemsize
• memoryview(c_arr).shape
• np.array(c_arr).dtype
• np.array(c_arr).shape
• np.asarray(c_arr).dtype
• np.asarray(c_arr).shape

Note that np.ctypeslib.as_array(carr) is as of 1.15 just an alias of np.asarray(carr). np.ctypeslib.as_array(ctr, shape) does a little bit extra, but nothing particularly useful.

[Strilanc]

This is in numpy 1.14

In [4]: print(
   ...:   memoryview(c_arr).itemsize,
   ...:   memoryview(c_arr).shape,
   ...:   np.array(c_arr).dtype,
   ...:   np.array(c_arr).shape,
   ...:   np.asarray(c_arr).dtype,
   ...:   np.asarray(c_arr).shape, sep='\n')
8
(1, 2)
int64
(1, 2)
int64
(1, 2)

Also some warnings were printed:

bin/ipython:4: RuntimeWarning: Item size computed from the PEP 3118 buffer format string does not match the actual item size.
  import re
bin/ipython:5: RuntimeWarning: Item size computed from the PEP 3118 buffer format string does not match the actual item size.
  import sys
numpy/core/numeric.py:492: RuntimeWarning: Item size computed from the PEP 3118 buffer format string does not match the actual item size.
  return array(a, dtype, copy=False, order=order)

This is in numpy 1.15

In [2]: print(
   ...:   memoryview(c_arr).itemsize,
   ...:   memoryview(c_arr).shape,
   ...:   np.array(c_arr).dtype,
   ...:   np.array(c_arr).shape,
   ...:   np.asarray(c_arr).dtype,
   ...:   np.asarray(c_arr).shape, sep='\n')

8
(1, 2)
int64
(2,)
int64
(2,)

And this is the warnings:

bin/ipython:4: RuntimeWarning: A builtin ctypes object gave a PEP3118 format string that does not match its itemsize, so a best-guess will be made of the data type. Newer versions of python may behave correctly.
  import re
bin/ipython:5: RuntimeWarning: A builtin ctypes object gave a PEP3118 format string that does not match its itemsize, so a best-guess will be made of the data type. Newer versions of python may behave correctly.
  import sys
python3.5/site-packages/numpy/core/numeric.py:501: RuntimeWarning: A builtin ctypes object gave a PEP3118 format string that does not match its itemsize, so a best-guess will be made of the data type. Newer versions of python may behave correctly.
  return array(a, dtype, copy=False, order=order)

Note that the so a best-guess will be made of the data type warning text only appears in 1.15.

[Strilanc]

I initially put 1.14 output by accident; you may have received a confusing email.

[eric-wieser]

I don't understand the warnings are being emitted there, but I do understand why the guess is failing:

>>> c_type_square = ctypes.c_uint8 * 3 * 3
>>> np.dtype(c_type_square)
dtype((('u1', (3,)), (3,)))  # a little janky, but array of arrays is a fair description
>>> c_type_col = ctypes.c_uint8 * 3 * 1
>>> np.dtype(c_type_col)
dtype(('u1', (3,))) # uh oh, singleton dimension is gone
>>> c_type_row = ctypes.c_uint8 * 1 * 3
>>> np.dtype(c_type_row)
dtype(('u1', (3,))) # same

This is caused by np.dtype((scalar, N)) not working when N == 1. There is an issue about that somewhere, but I think we might be stuck with it.

[eric-wieser]

Patched at #11652 - thanks for the bug report!