memoryview complain ctypes byte array are not native single character

[hct]

BPO	19803
Nosy	@skrah
Superseder	bpo-3132: implement PEP 3118 struct changes

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2013-11-29.11:07:39.335>
created_at = <Date 2013-11-26.20:54:09.824>
labels = ['interpreter-core', 'type-bug']
title = 'memoryview complain ctypes byte array are not native single character'
updated_at = <Date 2013-12-02.18:33:53.979>
user = 'https://bugs.python.org/hct'

bugs.python.org fields:

activity = <Date 2013-12-02.18:33:53.979>
actor = 'skrah'
assignee = 'none'
closed = True
closed_date = <Date 2013-11-29.11:07:39.335>
closer = 'skrah'
components = ['Interpreter Core']
creation = <Date 2013-11-26.20:54:09.824>
creator = 'hct'
dependencies = []
files = []
hgrepos = []
issue_num = 19803
keywords = []
message_count = 7.0
messages = ['204536', '204572', '204613', '204614', '204719', '205036', '205037']
nosy_count = 2.0
nosy_names = ['skrah', 'hct']
pr_nums = []
priority = 'normal'
resolution = 'duplicate'
stage = 'resolved'
status = 'closed'
superseder = '3132'
type = 'behavior'
url = 'https://bugs.python.org/issue19803'
versions = ['Python 3.3']

[hct]

I'm trying to create ctypes buffer to be used back and forth with C libraries, but I keep getting errors. I need to slice the buffer to cut out different pieces to work on, so I try to get a memoryview of the buffer. does the error message imply that c_ubyte, c_uint8, c_byte and c_char are not native single character types?

>>> memoryview(c_ubyte()).cast('B')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: memoryview: source format must be a native single character format prefixed with an optional '@'
>>> memoryview((c_ubyte*1024)()).cast('B')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: memoryview: source format must be a native single character format prefixed with an optional '@'
>>> memoryview(c_uint8()).cast('B')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: memoryview: source format must be a native single character format prefixed with an optional '@'
>>> memoryview((c_uint8*1024)()).cast('B')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: memoryview: source format must be a native single character format prefixed with an optional '@'
>>> memoryview((c_byte*1024)()).cast('B')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: memoryview: source format must be a native single character format prefixed with an optional '@'
>>> memoryview((c_char*1024)()).cast('B')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: memoryview: source format must be a native single character format prefixed with an optional '@'
>>> memoryview((c_char*1024)())[0]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NotImplementedError: memoryview: unsupported format (1024)<c
>>> memoryview((c_byte*1024)())[0]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NotImplementedError: memoryview: unsupported format (1024)<b
>>> memoryview((c_ubyte*1024)())[0]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NotImplementedError: memoryview: unsupported format (1024)<B
>>> memoryview((c_uint8*1024)())[0]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NotImplementedError: memoryview: unsupported format (1024)<B
>>>

[skrah]

Memoryview currently only knows the types from the struct module.

[hct]

this seems to disagree with the statement of "Memoryview currently only knows the types from the struct module."

why is memoryview aware of _pack_, a implementation detail of ctypes structures. is memoryview a collection of implementation for different types or a unique type on its own?

>>> import ctypes
>>> class B1(ctypes.Structure):
...     _fields_ = [( "data", c_uint8 * 256 ), ]
...     _pack_ = 1
...
>>> class B2(ctypes.Structure):
...     _fields_ = [( "data", c_uint8 * 256 ), ]
...
>>>
>>> a = B1()
>>> b = B2()
>>> memoryview( a ).cast( 'B' )
<memory at 0x01FFCDC0>
>>> memoryview( b ).cast( 'B' )
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: memoryview: source format must be a native single character format prefixed with an optional '@'
>>>

[hct]

more examples (using 64-bit integer vs 8-bit integer in the above example) to show that ctypes aren't being translated for memoryview properly. _pack_ is the only way to make memoryview handle ctypes properly

>>> import ctypes
>>> class B1(ctypes.Structure):
...     _fields_ = [( "data", ctypes.c_uint64 * 256 ), ]
...     _pack_ = 1
...
>>> class B2(ctypes.Structure):
...     _fields_ = [( "data", ctypes.c_uint64 * 256 ), ]
...
>>>
>>> a = B1()
>>> b = B2()
>>> memoryview( a ).cast( 'B' )
<memory at 0x01FFB030>
>>> memoryview( b ).cast( 'B' )
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: memoryview: source format must be a native single character format prefixed with an optional '@'
>>>

[skrah]

>>> class B1(ctypes.Structure):
...     _fields_ = [("data", ctypes.c_uint8 * 256), ]
...     _pack_ = 1 
... 
>>> a= B1()
>>> x = memoryview(a)
>>> x.format
'B'

In the first case the format is 'B', in the second case the
format is:

>>> x = memoryview(b)
>>> x.format
'T{(256)<B:data:}'

While the latter is probably a valid PEP-3118 format, it's
not implemented anywhere outside ctypes See bpo-3132.

[hct]

I wonder why "_pack_ = 1" has significant impact on the behaviour of memoryview if memoryview is not a sub class of ctypes structure. unless memoryview was specifically coded to understand ctypes structure, I don't see why "_pack_ = 1" should make any difference.

[skrah]

Memoryview sends a *request* to ctypes to fill in a Py_buffer struct according
to the buffer *protocol*. IOW, memoryview knows nothing about ctypes.

For some reason ctypes fills in 'B' as the format if _pack_ is set. I do not
know if that is intended, it could be a ctypes issue.