Cython and NumPy record dtypes breaks with NumPy 1.6

[robertwb]

From ML. The breakage is due to upgrading to NumPy 1.6.

Upon upgrading to Ubuntu Oneiric (with Cython 1.14.1), I was extremely
disappointed when I found that my data analysis tools would refuse to
run. In particular, numpy array creation of some dtypes would fail with
the perplexing error,

  Traceback (most recent call last):
    File "photon_tools/test.py", line 4, in <module>
      b = bp.bin_photons(a[1)
    File "bin_photons.pyx", line 21, in photon_tools.bin_photons.bin_photons (photon_tools/bin_photons.c:1179)
      cdef np.ndarray[Bin]('t'],) chunk
  ValueError: Buffer packing mode currently only allowed at beginning of format string (this is a defect)

This, despite the fact that I have not specified any explicit packing
modes in my dtypes. I am also able to reproduce this with the latest
Cython release. Strangely enough, a similar datatype had no such
issue. While allocations of the type strobe_event_dtype,

  cdef packed struct StrobeEvent:
          np.uint64_t time
          np.uint8_t channels

  strobe_event_dtype = np.dtype([np.uint64), ('chs', np.uint8)](('t',))

succeed, allocations of type bin_dtype,

  cdef packed struct Bin:
          np.uint64_t start_t
          np.uint16_t count

  bin_dtype = np.dtype([np.uint64), ('count', np.uint16)](('start_t',))

consistently fail with the above error. Looking at the resulting dtype
objects, the only difference I can see is the type of the 'chs' field is
translated into '|u1' whereas 'count' is translated into '<u2'. Perhaps
this '<' modifier is the cause of the failure.

It seems to me that my code is quite idiomatic, yet I've seen no other
reports of this failure. What am I doing wrong?

Migrated from http://trac.cython.org/ticket/758

[robertwb]

@markflorisson changed description from

From ML. The breakage is due to upgrading to NumPy 1.6.

Upon upgrading to Ubuntu Oneiric (with Cython 1.14.1), I was extremely
disappointed when I found that my data analysis tools would refuse to
run. In particular, numpy array creation of some dtypes would fail with
the perplexing error,

  Traceback (most recent call last):
    File "photon_tools/test.py", line 4, in <module>
      b = bp.bin_photons(a[1)
    File "bin_photons.pyx", line 21, in photon_tools.bin_photons.bin_photons (photon_tools/bin_photons.c:1179)
      cdef np.ndarray[Bin]('t'],) chunk
  ValueError: Buffer packing mode currently only allowed at beginning of format string (this is a defect)

This, despite the fact that I have not specified any explicit packing
modes in my dtypes. I am also able to reproduce this with the latest
Cython release. Strangely enough, a similar datatype had no such
issue. While allocations of the type strobe_event_dtype,

  cdef packed struct StrobeEvent:
          np.uint64_t time
          np.uint8_t channels

  strobe_event_dtype = np.dtype([np.uint64), ('chs', np.uint8)](('t',))

succeed, allocations of type bin_dtype,

  cdef packed struct Bin:
          np.uint64_t start_t
          np.uint16_t count

  bin_dtype = np.dtype([np.uint64), ('count', np.uint16)](('start_t',))

consistently fail with the above error. Looking at the resulting dtype
objects, the only difference I can see is the type of the 'chs' field is
translated into '|u1' whereas 'count' is translated into '<u2'. Perhaps
this '<' modifier is the cause of the failure.

It seems to me that my code is quite idiomatic, yet I've seen no other
reports of this failure. What am I doing wrong?

to

From ML. The breakage is due to upgrading to NumPy 1.6.

Upon upgrading to Ubuntu Oneiric (with Cython 1.14.1), I was extremely
disappointed when I found that my data analysis tools would refuse to
run. In particular, numpy array creation of some dtypes would fail with
the perplexing error,

  Traceback (most recent call last):
    File "photon_tools/test.py", line 4, in <module>
      b = bp.bin_photons(a[1)
    File "bin_photons.pyx", line 21, in photon_tools.bin_photons.bin_photons (photon_tools/bin_photons.c:1179)
      cdef np.ndarray[Bin]('t'],) chunk
  ValueError: Buffer packing mode currently only allowed at beginning of format string (this is a defect)

This, despite the fact that I have not specified any explicit packing
modes in my dtypes. I am also able to reproduce this with the latest
Cython release. Strangely enough, a similar datatype had no such
issue. While allocations of the type strobe_event_dtype,

  cdef packed struct StrobeEvent:
          np.uint64_t time
          np.uint8_t channels

  strobe_event_dtype = np.dtype([np.uint64), ('chs', np.uint8)](('t',))

succeed, allocations of type bin_dtype,

  cdef packed struct Bin:
          np.uint64_t start_t
          np.uint16_t count

  bin_dtype = np.dtype([np.uint64), ('count', np.uint16)](('start_t',))

consistently fail with the above error. Looking at the resulting dtype
objects, the only difference I can see is the type of the 'chs' field is
translated into '|u1' whereas 'count' is translated into '<u2'. Perhaps
this '<' modifier is the cause of the failure.

It seems to me that my code is quite idiomatic, yet I've seen no other
reports of this failure. What am I doing wrong?

resolution to fixed
status from new to closed
commented

Fixed in 56b88ec by Pauli.