ctypes cast and from_address cause crash on Windows 10

[MordechaiBotrashvily]

BPO	35326
Nosy	@eryksun
Files	test_ctypes_shmem.py: Code to reproduce the crash on Windows 10 1803 and python 3.6.2

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-11-27.17:11:07.173>
created_at = <Date 2018-11-27.11:00:03.143>
labels = ['ctypes', '3.7', 'invalid', 'type-crash']
title = 'ctypes cast and from_address cause crash on Windows 10'
updated_at = <Date 2018-11-28.17:26:00.836>
user = 'https://bugs.python.org/MordechaiBotrashvily'

bugs.python.org fields:

activity = <Date 2018-11-28.17:26:00.836>
actor = 'Mordechai Botrashvily'
assignee = 'none'
closed = True
closed_date = <Date 2018-11-27.17:11:07.173>
closer = 'eryksun'
components = ['ctypes']
creation = <Date 2018-11-27.11:00:03.143>
creator = 'Mordechai Botrashvily'
dependencies = []
files = ['47951']
hgrepos = []
issue_num = 35326
keywords = []
message_count = 3.0
messages = ['330503', '330539', '330618']
nosy_count = 2.0
nosy_names = ['eryksun', 'Mordechai Botrashvily']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue35326'
versions = ['Python 3.5', 'Python 3.6', 'Python 3.7']

[MordechaiBotrashvily]

Hi,

Using cast() or from_address() to convert from c_void_p (or integer) address to POINTER(c_ubyte) causes the interpreter to crash, when accessing the contents of the array.

The problem seems to happen when running the following combination:
Windows 10 @ 64 bit, version 1803 (OS build 17134.407) and python 3 64 bit, starting with 3.5 until 3.7 (the last that was tested).

The following code works fine on the same system using python 2.7.15 or python 3 until 3.4 (inclusive), 64 bit.
In addition, it works well also under Windows 7 64 bit and python 3.5/3.6/3.7 etc.

How to reproduce?

from ctypes import wintypes, windll, c_void_p, c_size_t, POINTER, c_ubyte, cast
# Define constants.
FILE_MAP_ALL_ACCESS = 983071
PAGE_READWRITE = 4
# Configure function arguments.
windll.kernel32.CreateFileMappingA.argtypes = [wintypes.HANDLE, c_void_p, wintypes.DWORD, wintypes.DWORD, wintypes.DWORD, wintypes.LPCSTR]
windll.kernel32.CreateFileMappingA.restype = wintypes.HANDLE
windll.kernel32.MapViewOfFile.argtypes = [wintypes.HANDLE, wintypes.DWORD, wintypes.DWORD, wintypes.DWORD, c_size_t]
windll.kernel32.MapViewOfFile.restypes = wintypes.LPVOID
# Open shared-memory.
handle = windll.kernel32.CreateFileMappingA(-1, None, PAGE_READWRITE, 0, 1024 * 1024, b'TestSHMEM')
# Obtain pointer to SHMEM buffer.
ptr = windll.kernel32.MapViewOfFile(handle, FILE_MAP_ALL_ACCESS, 0, 0, 1024 * 1024)
arr = cast(ptr, POINTER(c_ubyte))
# or
#arr = POINTER(c_ubyte).from_address(ptr)
arr[0]
# now crashes

Also note that changing the return type of MapViewOfFile as follows:
windll.kernel32.MapViewOfFile.restypes = wintypes.LPVOID
is changed to:
windll.kernel32.MapViewOfFile.restypes = POINTER(c_ubyte)
The contents of the buffer can be directly accessed without a problem.

Just in case I'm not using it properly with new python releases?

Thanks!
Moti.

[eryksun]

The default function result type is c_int, which truncates a 64-bit pointer to 32-bit. The attribute that needs to be set is singular restype, not plural restypes. Unfortunately ctypes objects have a dict, so you're not catching the typo in an obvious way.

Additional comments:

POINTER(c_ubyte).from_address(ptr) is not the same as cast(ptr, POINTER(c_ubyte)). The first one wrongly instantiates a pointer from the value at the ptr address instead of the address itself. If you want to use from_address, where ptr is a Python integer, a correct expression for this is POINTER(c_ubyte)(c_ubyte.from_address(ptr)).

I recommend using kernel32 = WinDLL('kernel32', use_last_error=True) instead of windll. This allows using ctypes.get_last_error() instead of kernel32.GetLastError(), which is especially more reliable when working in the REPL. It also avoids the problem with cached libraries on the global windll object, which can lead to prototype conflicts between libraries (e.g. if a script makes use of your library and another that uses the Windows API via ctypes, such as colorama), since libraries in turn cache function pointers.

[MordechaiBotrashvily]

Eryk, thanks :).
This really solved the problem.
Odd how it went unnoticed.