Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

memoryviews of arrays of structs fail to work in some cases #1407

Closed
robertwb opened this issue Jun 29, 2015 · 3 comments
Closed

memoryviews of arrays of structs fail to work in some cases #1407

robertwb opened this issue Jun 29, 2015 · 3 comments

Comments

@robertwb
Copy link
Contributor

As reported on the mailing list:

There seem to be some bugs regarding memoryviews of structured arrays
in Cython when the struct itself contains arrays. I have a Cython fix for
one of them (the most relevant to me), but first I would like to make sure
that these are really bugs and I am not doing anything stupid. This
question is related to
http://stackoverflow.com/questions/17239091/cython-memoryviews-from-array-of-structs,
but the answer there works only for char arrays. An array of doubles breaks
it again.

It seems to me that Cython implementation solves some special cases, but
not the general one. My proposed fix would be to insert

ndim = ctx->head->field->type->ndim;

after

/* Process the previous element */

if (__Pyx_BufFmt_ProcessTypeChunk(ctx) == -1) return NULL;

in Cython\Utility\Buffer.c

This fixes tests 2, 4 and 7, but not 8 and 9 (see below). I am not sure
about the details of what the code in Buffer.c is supposed to do and
whether my modification would break some other functionality, but my code
(not only the tests) works with this fix.

Or am I possibly simply missing "the right way" to handle arrays of structs
containing arrays in Cython? But in either case the compiler shouldn't
crash like in test 9...

Here is the code of my tests with and without numpy with results ranging
from "ok" to "compiler crash".
The numpy tests require these imports:

import numpy as np
cimport numpy as np
*Test 1 (OK)*
cdef packed struct My_Struct:
    double my_first_value
    double my_second_value
cpdef test():
    cdef My_Struct my_struct
    cdef My_Struct[my_view = <My_Struct[:1](:])>&my_struct

*Test 2 (FAILS)*
cdef packed struct My_Struct:
    double my_first_value
    double my_second_value*[test():
    cdef My_Struct my_struct
    cdef My_Struct[:](1]*
cpdef) my_view = <My_Struct[Expected 1 dimension(s), got 1*
*
*
*Test 3 (OK)*
cdef packed struct My_Struct:
    double my_first_value
    double my_second_value[*1*](:1]>&my_struct

*ValueError:)
cpdef test():
    cdef np.ndarray my_array = np.ndarray(1, 
dtype=np.dtype([('my_second_value','float64',*
1*)](('my_first_value','float64'),), align=False)
    cdef My_Struct[my_view = my_array

*Test 4 (FAILS)*
cdef packed struct My_Struct:
    double my_first_value
    double my_second_value[*2*](:])
cpdef test():
    cdef np.ndarray my_array = np.ndarray(1, 
dtype=np.dtype([('my_second_value','float64',*
2*)](('my_first_value','float64'),), align=False)
    cdef My_Struct[my_view = my_array

*ValueError: Expected 1 dimension(s), got 1*
*
*
*Test 5 (OK)*
cdef packed struct My_Struct:
    double my_first_value
    *char* my_second_value[2](:])
cpdef test():
    cdef np.ndarray my_array = np.ndarray(1, 
dtype=np.dtype([('my_second_value','*a*',2)](('my_first_value','float64'),), 
align=False)
    cdef My_Struct[my_view = my_array

*Test 6 (OK)*
cdef packed struct My_Array_Struct:
    double values[*1*](:])
cdef packed struct My_Struct:
    double my_first_value
    My_Array_Struct my_second_value
cpdef test():
    cdef np.ndarray my_array = np.ndarray(1, 
dtype=np.dtype([('my_second_value','float64',*
1*)](('my_first_value','float64'),), align=False)
    cdef My_Struct[my_view = my_array

*Test 7 (FAILS)*
cdef packed struct My_Array_Struct:
    double values[*2*](:])
cdef packed struct My_Struct:
    double my_first_value
    My_Array_Struct my_second_value
cpdef test():
    cdef np.ndarray my_array = np.ndarray(1, 
dtype=np.dtype([('my_second_value','float64',*
2*)](('my_first_value','float64'),), align=False)
    cdef My_Struct[my_view = my_array

*ValueError: Expected 1 dimension(s), got 1*

*Test 8 (FAILS)*
cdef packed struct My_Struct:
    double my_first_value*[1](:])*
    double my_second_value
cpdef test():
    cdef My_Struct my_struct
    cdef My_Struct[my_view = <My_Struct[:1](:])>&my_struct

*ValueError: Buffer dtype mismatch, expected 'double' but got end in 
'My_Struct.my_second_value'*

*Test 9 (COMPILER CRASH)*
cdef packed struct My_Array_Struct:
    double values[packed struct My_Struct:
    double my_first_value
    My_Array_Struct my_second_value*[1](2]
cdef)*
cpdef test():
    cdef My_Struct my_struct
    cdef My_Struct[my_view = <My_Struct[:1](:])>&my_struct

Migrated from http://trac.cython.org/ticket/853

@robertwb
Copy link
Contributor Author

scoder changed description from

As reported on the mailing list:

There seem to be some bugs regarding memoryviews of structured arrays
in Cython when the struct itself contains arrays. I have a Cython fix for
one of them (the most relevant to me), but first I would like to make sure
that these are really bugs and I am not doing anything stupid. This
question is related to
http://stackoverflow.com/questions/17239091/cython-memoryviews-from-array-of-structs,
but the answer there works only for char arrays. An array of doubles breaks
it again.

It seems to me that Cython implementation solves some special cases, but
not the general one. My proposed fix would be to insert

ndim = ctx->head->field->type->ndim;

after

/* Process the previous element */

if (__Pyx_BufFmt_ProcessTypeChunk(ctx) == -1) return NULL;

in Cython\Utility\Buffer.c

This fixes tests 2, 4 and 7, but not 8 and 9 (see below). I am not sure
about the details of what the code in Buffer.c is supposed to do and
whether my modification would break some other functionality, but my code
(not only the tests) works with this fix.

Or am I possibly simply missing "the right way" to handle arrays of structs
containing arrays in Cython? But in either case the compiler shouldn't
crash like in test 9...

Here is the code of my tests with and without numpy with results ranging
from "ok" to "compiler crash".
The numpy tests require these imports:

import numpy as np
cimport numpy as np

*Test 1 (OK)*
cdef packed struct My_Struct:
    double my_first_value
    double my_second_value
cpdef test():
    cdef My_Struct my_struct
    cdef My_Struct[my_view = <My_Struct[:1](:])>&my_struct

*Test 2 (FAILS)*
cdef packed struct My_Struct:
    double my_first_value
    double my_second_value*[test():
    cdef My_Struct my_struct
    cdef My_Struct[:](1]*
cpdef) my_view = <My_Struct[Expected 1 dimension(s), got 1*
*
*
*Test 3 (OK)*
cdef packed struct My_Struct:
    double my_first_value
    double my_second_value[*1*](:1]>&my_struct

*ValueError:)
cpdef test():
    cdef np.ndarray my_array = np.ndarray(1, 
dtype=np.dtype([('my_second_value','float64',*
1*)](('my_first_value','float64'),), align=False)
    cdef My_Struct[my_view = my_array

*Test 4 (FAILS)*
cdef packed struct My_Struct:
    double my_first_value
    double my_second_value[*2*](:])
cpdef test():
    cdef np.ndarray my_array = np.ndarray(1, 
dtype=np.dtype([('my_second_value','float64',*
2*)](('my_first_value','float64'),), align=False)
    cdef My_Struct[my_view = my_array

*ValueError: Expected 1 dimension(s), got 1*
*
*
*Test 5 (OK)*
cdef packed struct My_Struct:
    double my_first_value
    *char* my_second_value[2](:])
cpdef test():
    cdef np.ndarray my_array = np.ndarray(1, 
dtype=np.dtype([('my_second_value','*a*',2)](('my_first_value','float64'),), 
align=False)
    cdef My_Struct[my_view = my_array

*Test 6 (OK)*
cdef packed struct My_Array_Struct:
    double values[*1*](:])
cdef packed struct My_Struct:
    double my_first_value
    My_Array_Struct my_second_value
cpdef test():
    cdef np.ndarray my_array = np.ndarray(1, 
dtype=np.dtype([('my_second_value','float64',*
1*)](('my_first_value','float64'),), align=False)
    cdef My_Struct[my_view = my_array

*Test 7 (FAILS)*
cdef packed struct My_Array_Struct:
    double values[*2*](:])
cdef packed struct My_Struct:
    double my_first_value
    My_Array_Struct my_second_value
cpdef test():
    cdef np.ndarray my_array = np.ndarray(1, 
dtype=np.dtype([('my_second_value','float64',*
2*)](('my_first_value','float64'),), align=False)
    cdef My_Struct[my_view = my_array

*ValueError: Expected 1 dimension(s), got 1*

*Test 8 (FAILS)*
cdef packed struct My_Struct:
    double my_first_value*[1](:])*
    double my_second_value
cpdef test():
    cdef My_Struct my_struct
    cdef My_Struct[my_view = <My_Struct[:1](:])>&my_struct

*ValueError: Buffer dtype mismatch, expected 'double' but got end in 
'My_Struct.my_second_value'*

*Test 9 (COMPILER CRASH)*
cdef packed struct My_Array_Struct:
    double values[packed struct My_Struct:
    double my_first_value
    My_Array_Struct my_second_value*[1](2]
cdef)*
cpdef test():
    cdef My_Struct my_struct
    cdef My_Struct[my_view = <My_Struct[:1](:])>&my_struct

to

As reported on the mailing list:

There seem to be some bugs regarding memoryviews of structured arrays
in Cython when the struct itself contains arrays. I have a Cython fix for
one of them (the most relevant to me), but first I would like to make sure
that these are really bugs and I am not doing anything stupid. This
question is related to
http://stackoverflow.com/questions/17239091/cython-memoryviews-from-array-of-structs,
but the answer there works only for char arrays. An array of doubles breaks
it again.

It seems to me that Cython implementation solves some special cases, but
not the general one. My proposed fix would be to insert

ndim = ctx->head->field->type->ndim;

after

/* Process the previous element */

if (__Pyx_BufFmt_ProcessTypeChunk(ctx) == -1) return NULL;

in Cython\Utility\Buffer.c

This fixes tests 2, 4 and 7, but not 8 and 9 (see below). I am not sure
about the details of what the code in Buffer.c is supposed to do and
whether my modification would break some other functionality, but my code
(not only the tests) works with this fix.

Or am I possibly simply missing "the right way" to handle arrays of structs
containing arrays in Cython? But in either case the compiler shouldn't
crash like in test 9...

Here is the code of my tests with and without numpy with results ranging
from "ok" to "compiler crash".
The numpy tests require these imports:

import numpy as np
cimport numpy as np
*Test 1 (OK)*
cdef packed struct My_Struct:
    double my_first_value
    double my_second_value
cpdef test():
    cdef My_Struct my_struct
    cdef My_Struct[my_view = <My_Struct[:1](:])>&my_struct

*Test 2 (FAILS)*
cdef packed struct My_Struct:
    double my_first_value
    double my_second_value*[test():
    cdef My_Struct my_struct
    cdef My_Struct[:](1]*
cpdef) my_view = <My_Struct[Expected 1 dimension(s), got 1*
*
*
*Test 3 (OK)*
cdef packed struct My_Struct:
    double my_first_value
    double my_second_value[*1*](:1]>&my_struct

*ValueError:)
cpdef test():
    cdef np.ndarray my_array = np.ndarray(1, 
dtype=np.dtype([('my_second_value','float64',*
1*)](('my_first_value','float64'),), align=False)
    cdef My_Struct[my_view = my_array

*Test 4 (FAILS)*
cdef packed struct My_Struct:
    double my_first_value
    double my_second_value[*2*](:])
cpdef test():
    cdef np.ndarray my_array = np.ndarray(1, 
dtype=np.dtype([('my_second_value','float64',*
2*)](('my_first_value','float64'),), align=False)
    cdef My_Struct[my_view = my_array

*ValueError: Expected 1 dimension(s), got 1*
*
*
*Test 5 (OK)*
cdef packed struct My_Struct:
    double my_first_value
    *char* my_second_value[2](:])
cpdef test():
    cdef np.ndarray my_array = np.ndarray(1, 
dtype=np.dtype([('my_second_value','*a*',2)](('my_first_value','float64'),), 
align=False)
    cdef My_Struct[my_view = my_array

*Test 6 (OK)*
cdef packed struct My_Array_Struct:
    double values[*1*](:])
cdef packed struct My_Struct:
    double my_first_value
    My_Array_Struct my_second_value
cpdef test():
    cdef np.ndarray my_array = np.ndarray(1, 
dtype=np.dtype([('my_second_value','float64',*
1*)](('my_first_value','float64'),), align=False)
    cdef My_Struct[my_view = my_array

*Test 7 (FAILS)*
cdef packed struct My_Array_Struct:
    double values[*2*](:])
cdef packed struct My_Struct:
    double my_first_value
    My_Array_Struct my_second_value
cpdef test():
    cdef np.ndarray my_array = np.ndarray(1, 
dtype=np.dtype([('my_second_value','float64',*
2*)](('my_first_value','float64'),), align=False)
    cdef My_Struct[my_view = my_array

*ValueError: Expected 1 dimension(s), got 1*

*Test 8 (FAILS)*
cdef packed struct My_Struct:
    double my_first_value*[1](:])*
    double my_second_value
cpdef test():
    cdef My_Struct my_struct
    cdef My_Struct[my_view = <My_Struct[:1](:])>&my_struct

*ValueError: Buffer dtype mismatch, expected 'double' but got end in 
'My_Struct.my_second_value'*

*Test 9 (COMPILER CRASH)*
cdef packed struct My_Array_Struct:
    double values[packed struct My_Struct:
    double my_first_value
    My_Array_Struct my_second_value*[1](2]
cdef)*
cpdef test():
    cdef My_Struct my_struct
    cdef My_Struct[my_view = <My_Struct[:1](:])>&my_struct

commented

@FxIII
Copy link

FxIII commented Jun 2, 2019

import numpy as np

ctypedef struct test_pass_t:
        double nodes[2]
        int tag

cdef container_pass = np.ndarray(1,dtype=np.dtype([('nodes', ('f8', 2)), ('tag', ('i4',1))],align=True))

cdef test_pass_t [:] res_pass = container_pass
print (res_pass[0])

ctypedef struct test_fail_t:
        int nodes[2]
        int tag

cdef container_fail = np.ndarray(1,dtype=np.dtype([('nodes', ('i4', 2)), ('tag', ('i4',1))],align=True))

cdef test_fail_t [:] res_fail = container_fail
print (res_fail[0])```

@scoder
Copy link
Contributor

scoder commented May 1, 2020

Closing this since it seems to be covered by #3562.

@scoder scoder closed this as completed May 1, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants