Crash when subclassing ctypes.Union

[oriordan]

BPO	38368
Nosy	@vsajip, @ned-deily, @ambv, @zooba, @tirkarthi, @oriordan
PRs	bpo-38368: Added fix for ctypes crash when handling arrays in structs… #16589 [3.8] bpo-38368: Added fix for ctypes crash when handling arrays in structs… #16671 [3.7] bpo-38368: Added fix for ctypes crash when handling arrays in struct/unions. (GH-16589) #16672

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-10-09.05:53:36.985>
created_at = <Date 2019-10-04.08:36:01.907>
labels = ['3.8', '3.9', 'release-blocker', 'ctypes', '3.7', 'type-crash']
title = 'Crash when subclassing ctypes.Union'
updated_at = <Date 2019-10-15.07:44:07.757>
user = 'https://github.com/oriordan'

bugs.python.org fields:

activity = <Date 2019-10-15.07:44:07.757>
actor = 'ned.deily'
assignee = 'none'
closed = True
closed_date = <Date 2019-10-09.05:53:36.985>
closer = 'vinay.sajip'
components = ['ctypes']
creation = <Date 2019-10-04.08:36:01.907>
creator = 'oriordan'
dependencies = []
files = []
hgrepos = []
issue_num = 38368
keywords = ['patch', '3.7regression', '3.8regression']
message_count = 12.0
messages = ['353906', '353909', '353955', '353956', '353958', '353959', '353967', '354233', '354249', '354250', '354692', '354704']
nosy_count = 6.0
nosy_names = ['vinay.sajip', 'ned.deily', 'lukasz.langa', 'steve.dower', 'xtreak', 'oriordan']
pr_nums = ['16589', '16671', '16672']
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue38368'
versions = ['Python 3.7', 'Python 3.8', 'Python 3.9']

[oriordan]

Ran into Segfaults while trying to use pysnmp with 3.8.0rc1.
The code is running fine on 3.8.0b04.

$ python3.8
Python 3.8.0rc1 (default, Oct  2 2019, 14:15:18)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-36)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ctypes
>>> class in6_addr_U(ctypes.Union):
...     _fields_ = [
...         ('__u6_addr8', ctypes.c_uint8 * 16),
...         ('__u6_addr16', ctypes.c_uint16 * 8),
...         ('__u6_addr32', ctypes.c_uint32 * 4),
...     ]
...
Segmentation fault

$ docker run -it python:3.8.0rc1-slim
Python 3.8.0rc1 (default, Oct  2 2019, 23:38:42)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ctypes
>>> class in6_addr_U(ctypes.Union):
...     _fields_ = [
...         ('__u6_addr8', ctypes.c_uint8 * 16),
...         ('__u6_addr16', ctypes.c_uint16 * 8),
...         ('__u6_addr32', ctypes.c_uint32 * 4),
...     ]
...
$

The code is from here: https://github.com/etingof/pysnmp/blob/master/pysnmp/carrier/sockmsg.py#L47-L52

[tirkarthi]

I am adding 3.8 regression since the report says 3.8.0b4 works and segfaults with 3.8.0RC1.

[zooba]

Also crashes on Windows (x64 and x86), and on 3.7.5rc1, so this is likely in our code.

[zooba]

I get this assertion in a debug build:

Assertion failed: actual_type_index <= MAX_ELEMENTS, file c:\projects\cpython\modules\_ctypes\stgdict.c, line 718

[zooba]

Increasing MAX_ELEMENTS fixes it, but I'm not sure what other impacts there are from doing that.

+Vinay who added the array handling that's hitting the limit.

[zooba]

Given this limit can easily be hit by user code, I'd like to see it turned into a proper check with an exception (or a dynamic array) rather than just an assertion. We should not segfault here.

while (length > 0) {
    actual_types[actual_type_index++] = &edict->ffi_type_pointer;
    assert(actual_type_index <= MAX_ELEMENTS);
    length--;
}

[vsajip]

We should not segfault here.

Agreed, MAX_ELEMENTS was set to be an upper bound which shouldn't be hit. I'll investigate with OP's example data and see where the bug is.

[vsajip]

New changeset e8bedbd by Vinay Sajip in branch 'master':
bpo-38368: Added fix for ctypes crash when handling arrays in structs… (GH-16589)
e8bedbd

[vsajip]

New changeset d004a5b by Vinay Sajip in branch '3.8':
bpo-38368: Added fix for ctypes crash when handling arrays in structs/unions. (GH-16589) (GH-16671)
d004a5b

[vsajip]

New changeset 129c2b3 by Vinay Sajip in branch '3.7':
bpo-38368: Added fix for ctypes crash when handling arrays in structs/unions. (GH-16589) (GH-16672)
129c2b3

[ned-deily]

New changeset 1c61f2c by Ned Deily (Vinay Sajip) in branch '3.7':
bpo-38368: Added fix for ctypes crash when handling arrays in structs/unions. (GH-16589) (GH-16672)
1c61f2c

[ned-deily]

(fix released in 3.8.0 and 3.7.5)